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ABSTRACT 


*"  Client  satisfaction  was  considered  to  be  a  critical  indicator  of  the  effectiveness  of 
the  services  provided  by  the  Navy  Regiorfal  Data  Automation  Centers  (NARDACs). 
The  purpose  of  this  thesis  was  to  develop  the  means  to  measure  this  indicator. 

Interviews  of  twenty-eight  middle  management  clients  served  by  NARDACs  in 
four  geographical  regions  within  CONUS  were  conducted.  Forty-four  items  that 
influence  satisfaction  were  identified  belonging  to  eight  factor  dimensions  of  the 
client  satisfaction  domain.  The  Likert  Scale  methodology  was  employed  in  the 
construction  of  the  measurement  instrument.  The  instrument  was  reviewed  for 
completeness  of  items  and  logical  soundness  of  operations  by  three  independent 
groups  of  experts,  thus  achieving  content  validity.  A  client  satisfaction  index  is 
formulated  from  data  obtained  with  the  measurement  instrument. 

The  instrument  was  piloted  to  a  NARDAC  site  consisting  of  fifty-two  client 
organizations  using  self-administered  mail  surveys.  Evaluation  of  the  instrument 
was  performed  on  the  scored  results  of  twenty-six  pilot  respondents.  The  pilot 
results  were  subjected  to  empirical  tests  for  construct  validity  and  reliability.  The 
results  indicated  that  the  instrument  could  be  used  to  measure  client  satisfaction  after 
further  piloting  on  a  larger  (at  least  two  hundred  twenty)  sample  size. 
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When  you  can  measure  what  you  are  speaking  about,  and  express  it  in 
numbers,  you  know  something  about  it;  but  when  you  cannot  express 
it  in  numbers,  your  knowledge  is  of  a  meager  and  unsatisfactory  kind; 
it  may  be  the  beginning  of  knowledge,  but  you  have  scarcely,  in  your 
thoughts,  advanced  to  the  stage  of  science.  [Cited  in  Ref.  1 :  p.  89] 


This  observation  by  the  renown  British  physicist,  William  Thompson  (Lord 
Kelvin),  may  have  been  foremost  in  the  minds  of  the  Naval  Data  Automation 
Command  (NAVDAC)  corporate  management  during  the  development  of  a  long-term 
business  strategy  for  their  field  activities,  the  Navy  Regional  Data  Automation  Center 
(NARDACs).  The  steering  committee,  composed  of  the  NARDAC  Commanding 
Officers  and  top  management  in  NAVDAC,  was  established  as  the  NARDAC  Board 
of  Directors  and  chartered  to  address  strategic  planning  matters  and  issues  affecting 
the  NARDACs  as  a  result  of  a  1986  Coopers  and  Lybrand  study  of  Navy  Industrial 
Fund  (NIF)  activities.1 

One  of  the  issues  that  the  NARDAC  Board  of  Directors  identified  in  the 
corporate  long-range  business  plan  was  a  need  for  a  standardized  index  which  would 
be  used  to  measure  the  level  of  client  satisfaction  at  the  various  NARDACs.  The 
index  would  serve  the  purpose  of  providing  an  objective  evaluation  of  each 
NARDACs  performance  and  its  ability  to  deliver  quality  service  to  its  clients.  Also, 
as  a  measure  of  effectiveness,  the  index  would  be  the  cornerstone  of  their  business 


1  The  Secretary  of  the  Navy  (SECNAV)  contracted  Coopers  and  Lybrand  to  conduct 
a  management  analysis  of  all  activities  under  the  NIF  program.  The  purpose  of  the  study 
was  to  assess  the  NIF  program’s  effectiveness  and  to  find  any  areas  of  deficiency. 
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strategy  to  increase  the  NARDAC's  competitive  posture.  This  would  be 
accomplished  by  accurately  evaluating  their  current  level  of  client  satisfaction  to 
enable  the  establishment  of  a  base  index  from  which  management  can  take 
appropriate  actions  to  make  improvements. 

Presently,  a  standardized  measurement  of  effectiveness  that  empirically 
represents  user  satisfaction  among  those  clients  serviced  by  the  NARDACs  does  not 
exist.  Each  NARDAC,  operating  as  a  cost  center,  is  responsible  for  marketing  its 
services  to  activities  normally  located  within  its  geographical  area  of  cognizance.  As 
a  result,  the  NARDACs  currently  assess  client  satisfaction  using  different  methods, 
criteria,  standards  and  metrics.  This,  in  effect,  has  made  it  difficult  for  NAVDAC  to 
obtain  an  accurate  assessment  of  client  satisfaction  at  each  of  their  NARDACs. 

A.  EVOLUTION  OF  THE  REQUIREMENT 

I .  Background 

The  NARDACs  were  formed  as  a  result  of  the  Navy  Automated  Data 
Processing  (ADP)  Reorganization  study  and  implementation  plan  with  the  basic 
objective  to  improve  the  management  and  operation  of  the  Navy's  mission  support 
ADP  program  [Ref.  2],  These  field  activities  of  NAVDAC  were  established  to 
provide  non-tactical  ADP  technical  assistance  and  operational  support  to  Navy 
activities  ashore  and  afloat,  Department  of  Defense  (DOD)  components  and  other 
Federal  agencies.  There  are  nine  (9)  NARDACs  geographically  dispersed  within  the 
continental  United  States  (CONUS)  and  in  Hawaii  (See  Figure  1.1).  Although  each 
NARDAC  may  vary  in  the  ADP  support  it  can  provide,  major  functional  services  are 
offered  at  every  site  (See  Appendix  A). 
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2.  History  of  Operations 

Prior  to  fiscal  year  1984,  the  NARDACs  were  mission-funded  which 
meant  that  every  NARDAC  received  an  annual  allotment  of  appropriated  funds  to 
cover  its  costs  for  the  support  provided  to  its  clients.  Based  on  that  operational 
philosophy,  ADP  products  and  services  were  delivered  at  no  cost  to  the  clients. 
Because  this  ADP  support  was  perceived  by  the  clients  as  free,  there  was  little 
concern  on  their  part  for  costs,  only  the  timeliness  and  quality  of  the  requested 
support.  In  that  regard,  the  NARDACs'  primary  mission  was  focused  on  providing 
ADP  support  to  their  clients  within  the  planned  budgets.  The  need  to  formally 
measure  client  satisfaction,  therefore,  was  not  a  paramount  issue  at  the  time. 

3 .  Present-Day  Operations 

Since  fiscal  year  1984,  the  NARDACs’  cost  of  operations  has  been 
financed  under  the  Navy  Industrial  Fund  (NIF),  a  revolving  fund  from  which 
working  capital  funded  by  Congress  to  the  Department  of  Defense  is  allocated  to  the 
Navy  as  an  appropriation.  The  initial  funds  by  the  NIF  appropriation  to  the 
NARDACs  were  used  to  finance  the  costs  of  providing  ADP  services  ordered  by 
their  clients.  In  return,  the  products  furnished  and  services  rendered  by  the 
NARDACs  to  other  Navy  activities  and  government  agencies  are  performed  on  a  cost 
reimbursable  basis  at  standardized  fiscal  year  rates.1  It  is  therefore  the  responsibility 
of  the  client  to  submit  a  Program  Objective  Memorandum  (POM)  and  budget 


1  Standardized  fiscal  rates  refer  to  stabilized  rates  for  products  and  services  that  are 
fixed  for  an  entire  fiscal  year.  The  period  of  a  fiscal  year  is  from  1  October  to  30 
September, 
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accordingly  for  needed  ADP  services.1  The  clients  pay  the  NARDACs  through  the 

use  of  a  reimbursable  order  citing  their  own  appropriated  funds.  In  theory,  the 

payment  or  reimbursement  would  then  put  the  corpus  of  the  revolving  fund  back  to 

where  it  started.  The  financial  operations  of  the  NARDACs  are  comparable  to  that  of 

private  enterprises,  i.e.  working  capital  and  clients  are  required.  The  NARDACs  and 

other  NIF  activities,  however,  are  run  on  a  non-profit  basis. 

Working  with  a  NIF  activity  is  much  like  contracting  with  a  contractor 
—  the  only  significant  difference  is  that  the  NIF  activity  is  not  out  to 
make  a  profit  for  stockholders.  [Ref.  3:  p.  H22] 

Moreover,  the  importance  of  how  effective  a  NARDAC  operates  cannot  be 

understated.  The  less  effective  the  NARDAC,  the  higher  the  charge  for  services 

[Ref.  3:  p.  HI]. 

According  to  CDR  Charles  Taylor,  Commanding  Officer  of  NARDAC 
Pearl  Harbor,  the  change  to  NIF  was  made  for  two  major  reasons.  First,  in  partial 
emulation  of  enterprises  in  the  commercial  sector,  the  intention  was  to  discipline  and 
motivate  the  NARDACs  to  provide  the  best  possible  service  or  risk  going  out  of 
business.  The  second  major  reason  was  to  provide  a  measure  of  discipline  to  the 
users.  If  services  remained  free  to  the  users,  as  was  the  case  when  the  NARDACs 
were  under  mission  funding,  there  would  be  no  incentive  for  the  users  to  request 
only  those  services  critical  for  mission  accomplishment.  Together,  these  two  factors 
were  intended  to  guarantee  more  efficient  use  of  ADP  resources  in  the  Navy. 


1  The  POM  is  a  DOD  Planning,  Programming  and  Budgeting  System  (PPBS) 
document,  prepared  by  the  Services  to  the  Secretary  of  Defense,  which  expresses  total 
program  requirements  in  terms  of  force  structure,  manpower,  material  and  costs  to  satisfy 
assigned  functions  and  responsibilities  during  the  period  of  the  Five  Year  Defense 
Program.  [Ref  3:  p.  All] 
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4.  Impact  of  NIF  on  Operations 

It  was  clearly  obvious  to  the  NARDACs  that  the  transition  to  NIF  forced  a 
significant  change  in  their  way  of  doing  business.  NIF  had  introduced  new  factors 
into  the  NARDACs'  system  of  operations  which  had,  until  then,  only  applied  to 
comparable  organizations  in  the  commercial  sector.  The  principal  change  in  direction 
was  now  focused  on  the  clientele  from  whom  revenues  to  continue  operations  were 
obtained.  In  addition  to  providing  quality  ADP  support,  the  issue  of  operating  cost- 
effectively  and  to  remain  reasonably  affordable  for  the  clients,  became  of  paramount 
importance.  Moreover,  unlike  other  NIF  activities  who  maintain  a  quasi-monopoly 
over  the  products  and  services  they  provide,  the  NARDACs  were  now  forced  to 
compete  with  other  government  agencies,  as  well  as  the  private  sector,  to  provide 
non-tactical  ADP  support. 

Because  of  the  changes  driven  by  NIF,  client  satisfaction  has  become 
critical  for  the  NARDACs.  Since  all  services  are  provided  on  a  cost  reimbursable 
basis  and  all  operating  expenses  are  paid  for  by  revenues  generated  by  the  clients,  the 
NARDACs  must  maintain  a  positive  image  and  provide  quality  service  in  order  to 
retain  their  share  of  the  market.  A  client  satisfaction  index,  therefore,  could  serve  as 
a  tool  by  which  the  NARDACs  can  gauge  their  market  leverage  and  consequently 
make  whatever  adjustments  necessary  to  remain  competitive  and  financially 
operational. 

B.  OBJECTIVES 

The  objectives  of  this  thesis  are  to  identify  the  attributes  relating  to  client 
satisfaction  and  to  develop  a  standardized  methodology  for  measuring  and  analyzing 
these  attributes  through  the  construction  of  a  prototype  measurement  instrument  and 
client  satisfaction  index  for  subsequent  piloting  and  review  at  the  NARDACs. 
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C.  FOCAL  RESEARCH  QUESTIONS 

The  focal  issues  of  this  thesis  are: 

•  What  is  an  acceptable  definition  of  a  satisfied  client? 

•  What  attributes  should  be  used  to  construct  a  client  satisfaction  index  for  a 
NARDAC? 

•  How  should  the  data  be  gathered  and  in  what  form? 

•  What  metrics  should  be  used  to  measure  and  analyze  the  data  and  how  can 
these  metrics  be  validated? 

•  Can  a  microcomputer  be  used  for  data  analysis  and  for  maintaining 
information  on  customer  satisfaction?  If  so,  what  software  should  be  used? 

•  How  can  the  data  gathering  and  analysis  process  be  institutionalized  to  make  it 
part  of  an  ongoing  effort  to  meet  client  needs? 

D.  SCOPE,  LIMITATIONS  AND  ASSUMPTIONS 

The  scope  of  this  thesis  is  limited  to  the  development  of  a  standardized  prototype 
client  satisfaction  index  specifically  tailored  to  represent  an  empirical  measurement  of 
client  satisfaction  for  the  nine  NARDACs.  The  NARDACs  are  located  in  CONUS  at 
Naval  Air  Station  (NAS)  Alameda  -  for  NARDAC  San  Francisco,  NAS  North  Island 
-  for  NARDAC  San  Diego,  New  Orleans,  NAS  Pensacola,  NAS  Jacksonville,  NAS 
Norfolk,  Washington  Navy  Yard  -  for  NARDAC  Washington,  and  Newport;  and  in 
Hawaii  at  Naval  Station  Pearl  Harbor.  Each  NARDAC  has  a  unique  set  of  clients 
within  its  geographical  region.  The  research  focuses  primarily  on  those  attributes 
relating  to  client  satisfaction  as  ascertained  from  a  cross-section  of  NARDAC  clients. 

E.  ORGANIZATION  OF  THE  THESIS 

Chapter  II  presents  a  literature  review  pertinent  to  current  thinking  regarding 
client  satisfaction,  in  private  industry  and  in  government. 
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Chapter  III  describes  the  methodology  employed  in  the  development  of  a 
measurement  instrument  designed  as  the  basis  for  the  formulation  of  the  client 
satisfaction  index. 

Chapter  IV  presents  an  evaluation  of  the  validity  and  reliability  of  the  client 
satisfaction  questionnaire  as  a  measurement  instrument. 

Chapter  V  describes  the  construction  of  the  Client  Satisfaction  Index  (CSI)  and 
several  other  methods  of  analyzing  the  data  collected  from  the  questionnaire. 

Chapter  VI  presents  the  conclusions  of  the  study  and  recommendations  for 
implementation  and  further  research  .  This  chapter  concludes  the  thesis. 
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II.  LITERATURE  REVIEW 


A.  INTRODUCTION 

The  theme  of  this  year's  American  Marketing  Association's  annual  meeting  was 

"The  Customer  Is  King."  The  meeting  focused  on,  and  stressed  the  importance  of 

attracting  and  retaining  customers.  Many  corporations  and  management  consultants 

have  come  to  the  same  realization  that  the  key  to  maximizing  profits  is  maximizing 

customer  satisfaction  (e.g.,  [Ref.  4:  pp.  2-5],  [Ref  5:  pp.  13-16],  [Ref.  6:  p.  5]). 

During  the  past  five  years  of  financial  deregulation,  banking  strategists  have 

increasingly  concluded  that  one  of  a  bank's  most  important  assets  is  its  existing 

customer  base  [Ref.  7:  pp.  6,  57].  IBM  believes  that  satisfied  customers  are  the 

basis  of  their  continued  business  success  [Ref.  8].  In  the  words  of  Darryl 

Landvater,  President  of  Oliver  Wright  Video  Production,  Inc., 

Although  there  are  other  ways  to  get  an  additional  five  percent  of  sales 
to  the  bottom  line,  many  of  them  such  as  cutting  expenditures  for 
labor,  equipment,  and  research  and  development  are  both  painful  and 
potentially  damaging.  It's  easier  to  boost  sales  and  earnings  through 
superior  customer  service  and  at  the  same  time  improve  the  the  long¬ 
term  position  of  the  company.  [Ref.  9:  p.  86] 

Paul  Allaire,  President  of  Xerox,  has  recently  made  customer  satisfaction  it's 
"first  priority",  with  return  on  assets  (ROA)  and  increased  market  share  second  and 
third  respectfully  [Ref.  4:  p.  2].  In  a  memo  to  his  division  directors,  Allaire  says, 
"we  can  only  achieve  our  ROA  and  market  share  goals  by  satisfying  our  customers." 
Xerox's  new  focus  on  customer  satisfaction  was  the  basis  for  the  cover  story, 
"Customer  Satisfaction:  The  Big  Payoff'  in  Xerox’s  quarterly  corporate  publication 
Benchmark  [Ref.  4],  In  this  article,  five  leading  management  consultants  expound 


on  the  importance  of  satisfying  the  customer  and  its  payoff  in  increased  profits.  In 

the  article,  Robert  Waterman,  Jr.,  co-author  of  In  Search  of  Excellence,  cited  a  study 

from  the  Strategic  Planning  Institute  in  Cambridge,  Massachusetts. 

Business  sectors  with  a  higher  service  content  have  a  higher  return  on 
investment.  The  institute,  which  ranked  similar  companies  in  order  by 
the  relative  quality  of  their  products  and  services,  found  that  the 
highest  rated  company  in  each  area  had  a  return  on  investment  100 
percent  higher  than  the  lowest-ranked  company.  [Ref.  4:  p.  3] 

Christopher  Lovelock,  a  former  Harvard  professor  and  principal  of  Christopher 
Lovelock  and  Associates,  states  that  the  key  to  improving  customer  satisfaction  "...is 
with  research  to  measure  what  characteristics  of  services  or  products  are  important  to 
users  as  well  as  how  customers  perceive  the  goals  provided  by  the  company."  [Ref. 
4:  p.  3] 

Tom  Peters,  co-author  of  In  Search  of  Excellence  and  A  Passion  for  Excellence, 
believes  that  the  qualitative  aspects  of  business  must  be  quantified.  He  believes  in  the 
old  adage  "What  gets  measured  gets  done."  This  philosophy  is  applied  rigorously 
within  his  organization  and  at  his  executive  seminars  in  the  following  manner. 


In  the  customer  arena,  we  believe  that  regular  quantitative  measurement 
of  customer  satisfaction  provides  a  much  better  lead  indicator  of  future 
organizational  health  than  does  profitability  or  market-share  change. 

We  suggest  monthly  measurement.  Further,  we  urge  participants  to 
make  the  level  of  customer  satisfaction  the  primary  basis  for  incentive 
compensation  and  annual  performance  evaluation  for  virtually  every 
person  at  every  level  in  every  function  throughout  the  organization. 

[Ref.  10:  p.  El] 

Clearly,  customer  satisfaction  is  an  issue  which  is  receiving  considerable 
attention  in  the  corporate  community  and  recently  it  has  been  gaining  momentum  in 
the  computer  services  industry.  The  purpose  of  this  literature  review  is  to  survey 
some  of  the  recent  research  that  has  been  conducted  in  the  area  of  customer 
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satisfaction  with  computer  services  and  the  instruments  which  were  developed  to 
quantify  and  measure  satisfaction.  The  literature  review  is  divided  into  three 
sections.  The  first  section.  Measures  of  Computer  User  Satisfaction,  is  a  review  of 
generic  measures  which  could  be  used  at  any  computer  services  organization  or 
management  information  system/data  processing  (MIS/DP)  department.  The  second 
section,  Survey  of  Current  Industry  Measures,  is  a  review  of  current  measures  of 
customer  satisfaction  used  in  the  computer  industry.  And  the  last  section. 
Department  of  Defense  Measures,  is  a  brief  look  at  what  kinds  of  measures  of 
customer  satisfaction  are  being  used  within  the  Department  of  Defense. 

B .  MEASURES  OF  COMPUTER  USER  SATISFACTION 

The  largest  initial  obstacle  in  conducting  this  literature  review  was  finding 
information  related  to  the  measurement  of  customer  satisfaction  at  a  computer 
services  organization.  The  problem  was  the  word  "customer".  The  computer 
industry  and  MIS/DP  departments,  in  many  instances,  still  refers  to  its  customers  as 
users  instead  of  valuable  customers  [Ref.  1 1:  p.  286].  According  to  W.H.  Inmon, 
"Universally,  data  processing  exists  as  a  service  organization  within  the  company. 
The  service  is  for  the  user,  and  the  satisfaction  of  the  user  ultimately  determines  the 
success  or  failure  of  data  processing."  [Ref.  12:  p.  224] 

There  have  been  several  studies  conducted  recently  on  the  concept  of  computer 
user  satisfaction  or  user  information  satisfaction  (UIS).  In  the  words  of  Iivari,  UIS 
refers  to  "...a  cluster  of  concepts  that  imply  the  assessment  of  information  systems  or 
information  services  in  the  user's  subjective  terms."  [Ref.  13:  p.  57]  The 
foundations  of  UIS  are  based  on  the  research  by  Cyert  and  March  [Ref.  14],  The 
essential  concept  is  "...that  an  information  system  which  meets  the  needs  of  its  user 
will  reinforce  satisfaction  with  that  system.  If  the  system  does  not  provide  the 
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needed  information,  the  user  will  become  dissatisfied  and  look  elsewhere  [Ref.  15: 
p.786] 

One  of  the  first  measures  of  UIS  related  to  the  total  MIS/DP  function  in  an 
organization  which  has  received  considerable  attention  was  developed  by  S.  W. 
Pearson  [Ref.  16).  During  his  research,  Pearson  compiled  a  list  of  thirty-six  factors 
relating  to  computer  user  satisfaction  based  on  a  literature  review  of  twenty-two 
studies  of  computer-user  interactions.  These  factors  were  then  reviewed  by  three 
DP  professionals  for  completeness  and  accuracy.  As  a  result  of  the  review,  two 
additional  factors  were  added  to  the  list.  Next,  the  list  of  thirty-eight  factors  was 
compared,  using  a  critical  incident  analysis  technique,  to  taped  interview  responses 
from  thirty-two  middle  manager  users  in  eight  different  organizations.  This  step 
concluded  with  the  addition  of  one  factor,  making  a  total  of  thirty-nine  distinct 
factors.  These  factors  were  then  incorporated  into  a  questionnaire  which  utilized  the 
semantic  differential  technique.  Each  factor  is  measured  by  three  separate  scales 
using  seven  intervals  from  negative  to  positive  responses.  The  first  scale  was  a 
measure  of  one's  perception  by  rating  four  bipolar  adjective  pairs.  The  second  scale 
was  a  satisfactory  -  unsatisfactory  pair  and  the  third  scale  measured  the  importance  of 
the  factor  to  the  user.  The  importance  rating  is  used  as  a  weighting  factor  for  the 
overall  satisfaction  score.  The  description  of  the  scoring  method  can  be  found  in 
Reference  17.  The  questionnaire  was  empirically  tested  for  validity  and  reliability 
using  the  same  thirty-two  middle  managers  previously  interviewed.  The  instrument 
was  found  to  be  reliable  and  valid,  based  on  the  twenty-nine  returned  questionnaires. 
See  Reference  17  for  the  results  of  the  validity  and  reliability  evaluation  of  the 
questionnaire. 


A  study  by  Ives,  Olson  and  Baroudi  was  a  continuation  of  the  Pearson  study 
with  the  emphasis  to  reduce  the  length  of  the  overall  measure  and  reinforce  the 
validity  of  the  instrument  with  more  extensive  testing  [Ref.  15].  The  results  of  their 
study  reduced  Pearson's  thirty-nine  original  factors  down  to  twenty-one  factors  with 
greater  validity  using  a  larger  sample  size.  Description  of  their  results  can  be  found 
in  [Ref.  15].  Ives,  Olson,  and  Baroudi's  work  was  based  on  the  following 
definition  of  UIS:  "...the  extent  to  which  users  believe  the  information  system 
available  to  them  meets  their  information  requirements." 

Mathew  &  Co.,  a  Data  Processing  Management  consulting  firm,  has  developed  a 
reporting  system  for  measuring  DP-user  satisfaction  called  "How’re  We  Doing?" 
[Ref.  18:  p.  10]  The  system  requires  users  to  annotate  in  a  log,  during  a  one  month 
period,  any  problems  with  data  processing  based  on  nine  "performance-related 
criteria."  At  the  end  of  the  month,  the  data  from  the  log(s)  are  processed  by  Mathew 
&  Co.  which  supplies  the  client  with  DP  user  satisfaction  analyses  in  several 
different  categories.  The  system  is  based  on  exception  reporting  (i.e.,  only  problems 
with  data  processing  are  reported).  The  nine  criteria  are  timeliness  of  output,  quality 
of  output,  online  availability,  response  time,  systems  development  schedules, 
response  to  problems,  and  attitude  and  cooperativeness.  Mathew  &  Co.  provides  its 
clients  with  reports  on  DP-user  trends  for  the  current  month  and  trends  over  periods 
of  three  months,  six  months  and  twelve  months. 

C.  SURVEY  OF  CURRENT  INDUSTRY  MEASURES 

The  banking  industry  recently  began  using  a  customer  service  index  to  measure 
and  manage  customer  service  [Ref.  7].  One  index  is  based  on  twenty  critical 
attributes  of  "Good  Customer  Service”  according  to  a  nationwide  survey  of  bank 
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customers.  Bank  customers  were  asked  to  score  all  twenty  attributes  of  their 
financial  institution  on  a  six-  point  scale,  (6  =  excellent).1 

Keeping  the  customer  satisfied  with  quality  products  and  services  is  the 
challenge  changing  the  face  of  the  automotive  industry  also.  The  automotive  industry 
uses  a  consumer  satisfaction  index  to  evaluate  consumer  demands  (e.g.,  increasing 
showroom  hours).  [Ref.  19] 

Sitmar  Cruises  attributes  its  highest  rate  of  repeat  passengers  of  any  major  cruise 

line  to  its  "intensive  customer  research"  using  a  16-page  customer  survey  [Ref.  4:  p. 

4].  Benjamin  French  believes  random  surveys  are  the  best  way  of  determining  the 

effectiveness  of  an  organization's  customer  satisfaction  program. 

A  brief  questionnaire  and  accompanying  letter  of  explanation  are 
simple  to  prepare  and  provide  an  inexpensive  way  of  taking  the  pulse 
of  your  customers.  In  addition  to  providing  you  with  data  on  your 
effectiveness,  the  survey  also  conveys  your  concern  for  solving 
customer's  problems  and  gives  you  another  opportunity  to  restore  their 
faith  in  your  company  and  its  products  or  services.  [Ref.  20:  p.  1 1 1  ] 

However,  James  Carman,  Professor  of  Business  Administration  at  the  Graduate 
School  of  Business,  University  of  California  at  Berkeley,  argues  that  surveys  must 
be  interpreted  carefully  because  it  is  difficult  for  some  customers  to  evaluate  the 
quality  of  the  service  they  received  if  they  are  lacking  knowledge  in  the  field.  [Ref. 
4:  p.  5] 

IBM  distributes  a  survey  to  every  one  of  their  customers  at  least  once  a  year  in 
order  to  directly  ascertain  information  relating  to  customer  satisfaction  and  to  improve 
their  business  relationship.  The  corporate  offices  send  out  periodic  surveys  to  clients 
requesting  their  service  perceptions.  This  is  done  independendy  from  the  regional 


1  The  index  was  developed  by  Financial  Products  Group,  a  Chicago-based  consulting 
firm  serving  the  financial  services  industry,  with  the  assistance  of  Market  Facts,  Inc. 
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offices.  In  fact,  regional  offices  don’t  even  know  when  this  occurs.  The  survey  is 
divided  into  eight  major  categories:  quality  of  products,  hardware  maintenance  and 
support,  systems  management,  marketing  support,  technical  support,  education, 
telephone  coverage,  and  overall  rating.  The  survey  is  scored  on  a  five-point  Liken 
Scale  (very  satisfied  to  very  dissatisfied)  with  a  block  for  no  opinion.  In  addition  to 
the  survey,  there  is  a  corporate-wide  complaint  system  where  client  complaints  are 
assigned  to  a  high-level  manager  for  investigation  and  follow-up.  [Ref.  8] 

At  Electronic  Data  Systems  Corporation  (EDS),  they  believe  feedback  from  the 
customer  is  an  important  step  in  maintaining  a  high  level  of  customer  satisfaction. 
EDS  North  American  Commercial  Group  uses  an  extensive  series  of  questionnaires 
as  part  of  their  Quality  Enhancement  Program  (QEP)1  to  keep  abreast  of  their 
customers  needs  and  to  continue  to  improve  services  and  products.  [Ref.  21  ] 

D.  DEPARTMENT  OF  DEFENSE  MEASURES 

The  Philadelphia  Naval  Shipyard,  a  NIF-funded  activity,  does  not  use  any 
proactive  initiatives  to  ascertain  customer  satisfaction.  Their  criteria  for  measuring 
performance  is  based  on  adhering  to  project  budget  and  schedule  constraints2.  The 
Military  Airlift  Command  (MAC),  an  Air  Force  industrial  funded  activity,  does  not 
actively  solicit  information  from  their  customers  regarding  customer  satisfaction 
either.  Instead,  suggestion/comment  forms  are  made  available  to  MAC  customers.3 


1  The  contents  of  the  questionnaire  and  the  specifics  of  the  QEP  implementation 
policies  and  procedures  are  proprietary  information  of  EDS  North  American  Commercial 
Group  and  will  not  be  disclosed  in  this  thesis. 

2  Interview  with  Mr.  Monteleon,  Management  Planning  Division,  Philadelphia  Naval 
Shipyard,  Philadelphia,  Pennsylvania,  5  February  1988. 

3  Interview  with  CDR  Jordan,  USN,  Navy  Liaison  Officer,  Military  Airlift  Command 
Headquarters,  Scott  Air  Force  Base,  5  February  1988. 
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The  Navy  Public  Works  Centers  (PWC),  which  are  NIF-funded  activities,  have 
recently  initiated  a  customer  satisfaction  evaluation  program  [Ref.  22].  The  program, 
"Customer  Evaluation  of  Support  Provided  by  PWC",  consists  of  a  fifty-item 
questionnaire  divided  into  six  categories.  The  respondent  is  requested  to  rate  the 
quality  of  service  provided  by  the  PWC  using  a  seven-point  scale.  The  questionnaire 
is  scored  in  the  following  manner: 

a.  For  customer  responses  where  the  "X"  is  placed  in  blocks  6  or  7, 
score  2  points; 

b.  For  customer  responses  where  the  "X"  is  placed  in  blocks  3,  4,  or 
5,  score  1  point; 

c.  For  customer  responses  where  the  "X"  is  placed  in  blocks  1  or  2, 
score  0  points. 

The  best  possible  score  is  100  points.  To  determine  the  annual  grade 
from  all  of  the  customers  who  received  a  questionnaire,  add  the  total 
questionnaire  scores  and  divide  by  the  total  number  of  questionnaires 
that  were  completed  in  the  year.  The  total  score  is  weighted  by 
customer  size  (i.e.  four  surveys  from  large  customers,  two  from 
medium-  sized  customers  and  one  each  from  all  others.)  The  result  is  a 
summarized  customer  evaluation  of  the  support  provided  by  the  PWC 
in  the  past  year. 

We  found  no  published  studies  performed  by  the  Department  of  the  Navy  or  the 
Department  of  Defense  in  the  area  of  customer  or  user  satisfaction  relating  to 
computer  services  organizations  or  MIS/DP  departments.1 


1  Searches  were  made  through  the  Defense  Logistics  Studies  Information  Exchange 
(DLSIE),  Defense  Technical  Information  Center  (DTIC),  and  DIALOG  Information 
Services,  Inc. 
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III.  QUESTIONNAIRE  DEVELOPMENT 


The  corporate  heads  of  the  NARDACs  identified  the  critical  requirement  for  a 
client  satisfaction  index,  thus  setting  the  stage  for  the  next  step  of  our  research.  We 
needed  to  identify  a  logical  process  to  follow  in  order  to  arrive  at  an  index  that 
accurately  represents  the  client's  satisfaction  with  the  NARDAC's  performance  in 
providing  products  and  services.  To  represent  that  state  of  satisfaction  as 
unequivocally  as  possible,  it  was  imperative  that  the  theme  of  our  methodology  be 
based  on  the  inputs  from  actual  NARDAC  clients  and  their  composite  view  of  client 
satisfaction.  This  was  important  because  it  ensured  that  the  fundamental  building 
blocks  for  the  construction  of  a  meaningful  index  were  obtained  directly  from 
representative  samples  of  the  source.  By  using  this  approach,  we  felt  that  our 
methodology  would  be  sound  in  terms  of  face  validity  and  the  resulting  product 
would  be  realistic  and  more  readily  accepted  by  the  NARDACs  and  their  client 
community. 

From  a  study  by  Pearson,  the  measurement  of  client  satisfaction  mirrors  the 
measurement  of  an  individual’s  attitude  or  evaluative  feelings  toward  some  feature  of 
the  subjects  of  interest  [Ref.  16].  There  are  various  ways  to  collect  data  for 
measuring  a  client's  attitude.  One  basic  means  is  by  interrogation  which  involves 
getting  people  to  answer  questions,  either  in  person  or  through  telephone  interviews. 
Another  interrogation  technique,  more  widely  used  for  measuring  a  person’s  attitude, 
involves  an  instrument,  in  most  cases  a  questionnaire,  which  is  one  of  the 
fundamental  tools  used  in  survey  research.  The  reason  for  the  popularity  of  the 
survey  research  method  is  due  to  the  numerous  benefits  it  offers. 


•  It  provides  a  method  for  empirical  verification  of  data. 

•  The  data  gathered  by  this  method  becomes  a  source  of  information  which  can 
be  conveniently  stored  through  automated  methods  and  can  be  analyzed  shortlv 
after  collection. 

•  The  responses  can  be  coded  into  a  standardized  form  for  recording  in  a 
quantitative  manner;  therefore  a  standardized  measurement  that  is  consistent 
across  respondents  is  achieved  and  lends  itself  to  the  development  of  an  index. 

•  Surveys  can  be  administered  fairly  readily  by  an  implementing  organization  (a 
NARDAC)  and  can  be  flexibly  designed  with  minimum  impact  on  the 
respondents'  (the  clients')  time.  ([Ref.  23]  and  [Ref.  24]) 

It  is  because  of  these  advantages  and  its  applicability  to  the  measurement  of  client 

satisfaction  that  the  survey  research  approach  was  selected  for  this  study. 

A.  BACKGROUND 

During  the  preliminary  stages  of  the  research,  background  information  on  the 
NARDACs  was  gathered  in  order  to  obtain  an  understanding  of  their  organizational 
structure  and  hierarchy,  corporate  culture  and  environment,  products  and  services, 
and  client  information.  This  background  information  was  obtained  during  site  visits 
at  NARDAC  San  Francisco,  NARDAC  Washington  and  NAVDAC.  Further  data 
was  received  from  NARDAC  Pearl  Harbor  and  NARDAC  San  Diego.  Additionally, 
the  marketing  representative  at  Honeywell  was  interviewed  to  see  how  client 
satisfaction  was  viewed  from  the  industry  perspective.  We  also  reviewed  literature 
on  customer  satisfaction  and  found  current  trends  in  the  field,  particularly  in  service- 
oriented  industries. 

At  the  same  time,  we  were  given  information  on  existing  customer  satisfaction 
methods  employed  by  the  NARDACs  and  Public  Works  Centers  (PWC).  Primarily, 
the  methods  used  by  these  communities  are  informal  and  based  on  feedback  from  the 
client,  usually  obtained  during  personal  visits  or  phone  conversation.  The  PWCs 
and  several  NARDACs  employed  more  formal  methods,  using  questionnaires  that 
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were  developed  in-house,  as  part  of  their  marketing  efforts.  Some  of  these  methods, 
extracted  from  a  summary  of  findings  [Ref.  25],  are  listed  below. 


•  NARDAC  Jacksonville  developed  a  client  assessment  form  which  is  used 
during  quarterly  client  visits.  This  assessment  seems  to  be  the  most  formal 
means  employed  to  gather  client  satisfaction  information  from  the  client's 
perspective. 

•  NARDAC  San  Diego  identified  specific  client  satisfaction  indices  with  it’s 
mainframe  clients.  These  indices  are:  "95%  products  on  time"  and  "98%  on¬ 
line  availability  during  prime  time  and  off-shift  hours."  Prime  time  availability 
is  particularly  significant  to  its  major  client,  the  Naval  Aviation  Depot.  These 
indices  are  assessed  and  reported  during  regular  client  meetings. 

•  NARDAC  San  Francisco  recently  developed  an  information  form  which  will 
be  used  by  client  relations  personnel  when  visiting  clients.  A  portion  of  this 
form  provides  clients  the  opportunity  to  express  satisfaction  with  or  concerns 
about  services. 

•  NARDAC  Pensacola  and  NARDAC  San  Diego  use  written  reports  of  visits  or 
phone  contacts  to  identify  client  problems  or  concerns.  These  reports  also 
provide  vehicles  to  convey  the  client's  satisfaction  with  services. 

•  NARDAC  Washington  developed  a  quarterly  rating  report  which  is  filled  out 
by  NARDAC  Washington  department  directors.  This  report  provides  the 
department  directors'  expectation  of  a  client’s  response  to  various  service 
satisfaction  questions. 


Because  of  the  wide  diversity  in  methods  used  among  the  NARDACs,  it  is 
understandable  why  the  NARDAC  Board  of  Directors  recognized  the  need  for  a 
standardized  method  of  obtaining  critical  client  satisfaction  information. 

B .  FACTOR  IDENTIFICATION 

Using  the  information  gathered  from  interviews  with  NARDAC  management 
personnel,  relevant  organizational  documents  and  literature  searches,  we  generated  a 
list  of  the  major  products  and  services  that  the  NARDAC  provides  to  its  customers. 
From  this  list,  an  initial  set  of  factors,  which  was  seen  as  measuring  the  NARDACs’ 
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performance  in  delivering  products  and  services,  was  compiled.  This  initial  set 
consisted  of  the  following  nine  (9)  factors: 

•  Computer  capabilities 

•  Ease  of  obtaining  services 

•  System  design  and  programming  expertise 

•  Technical  support 

•  ADP  consultation  and  assistance 

•  Training 

•  Cost  of  ADP  services 
«  Customer  support 

•  Timeliness  and  quality  of  products  and  services  provided 

In  contrast,  the  factors  used  in  existing  NARDAC  questionnaires  [Ref.  25]  included: 

•  Response  to  client  requests  (the  quality  of  response  and  the  rapidity  of 
response) 

•  Thoroughness  of  information  conveyed  to  clients 

•  NARDAC  service  accessibility  to  clients 

•  Cost 

•  Perception  of  the  reliability  of  NARDAC  personnel 

•  Non-recurrence  of  problems  and  willingness  to  refer  other  organizations  to 
NARDAC  for  computing  services 

C .  CONTENT  VALIDATION 

In  order  to  validate  the  accuracy  of  our  initial  set  of  factors  and  thereby  establish 
content  validity,  a  method  was  needed  which  was  expedient,  allowed  for  personal 
interaction  with  existing  clients  to  verify  the  assumptions  made  in  compiling  the  list 
of  factors,  and  permitted  the  opportunity  to  identify  other  aspects  of  the  NARDAC- 
client  relationship  which  may  have  been  overlooked  or  omitted  during  previous 
interviews  with  NARDAC  personnel  and  documentation  review.  It  was  decided  then 
that  interviews  with  a  representative  group  of  active  NARDAC  clients  would 


accomplish  this  objective.  A  series  of  interviews  with  clients  served  by  the 
NARDACs  would  be  conducted  within  research  limitations. 

1 .  Interview  Procedures 

Because  of  time  constraints  and  a  ceiling  on  travel  expenses,  it  was 
necessary  to  set  criteria  for  determining  our  interview  sample.  This  criteria  included 
location,  types  of  client  organizations  to  interview,  who  within  the  client  organization 
to  interview  and  number  of  interviews  to  conduct. 

The  first  and  foremost  criterion  was  location.  We  wanted  to  focus  on 
regions  having  a  large  concentration  of  Navy  activities  with  differing  ADP  support 
requirements  and  where  major  NARDAC  installations  supporting  these  Navy 
commands  offered  a  wide  variety  of  computer  services.  The  East  Coast  and  West 
Coast  regions  appeared  to  satisfy  this  criterion  and  offered  the  best  representative 
cross-section  of  respondents.  The  clients  to  be  selected  for  interview  were  those 
supported  by  NARDAC  San  Francisco  and  NARDAC  San  Diego  on  the  West  Coast 
and  NARDAC  Washington  and  NARDAC  Norfolk  on  the  East  Coast. 

The  next  criterion  was  types  of  clients  to  consider  for  the  interviews.  Since 
the  objective  in  this  phase  of  the  research  was  to  obtain  information  on  what  factors 
affect  or  influence  client  satisfaction,  we  wanted  to  interview  current/active  clients 
who  maintained  some  frequency  of  interaction  with  the  NARDAC  providing  them 
services.  One  way  of  determining  the  relative  degree  of  interaction  was  to  examine 
the  amount  of  revenue  generated  from  the  client  during  the  last  fiscal  year.  This  data 
was  readily  available  to  the  Client  Liaison  Officer  at  each  NARDAC  who  compiled  a 
ranked  listing  of  clients  (by  revenues  earned)  from  which  candidate  clients  were 
selected.  Although  interviews  with  previous  clients  who  had  elected  not  to  renew 
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services  with  the  NARDAC  were  considered,  we  were  unable  to  include  that  client 
category  under  this  criterion  because  of  time  constraints. 

Another  criterion  used  in  narrowing  the  sample  size  of  clients  for  the 
interview  was  selection  of  the  person  in  the  client  organization  with  whom  to  conduct 
the  interview.  We  determined  that  the  best  candidate  to  satisfy  this  criterion  was  the 
individual  designated  by  the  client  organization  as  the  command’s  official  liaison  or 
point  of  contact  (POC)  with  the  NARDAC.  This  individual  would  usually  belong  in 
middle  management,  serving  as  the  activity’s  ADP  Officer.  Since  POC  information 
normally  resided  alongside  the  client  organization  data,  this  information  was 
maintained  and  kept  current  by  the  NARDAC's  Client  Liaison  Officer. 

The  last  criterion,  number  of  interviews  to  conduct,  was  constrained  by  a 
number  of  factors: 

•  Number  of  days  available  to  conduct  the  interviews 

•  Availability  of  the  clients  for  the  interview 

•  Length  of  the  interview 

•  Travel  time  window  required  between  interviews  (during  normal  working 
hours) 

In  most  cases,  the  Client  Liaison  Officer  of  the  NARDAC  was  in  a  better  position  to 
juggle  these  constraints  and  therefore  coordinated  appointments  with  the  clients  and 
arranged  the  interview  schedule.  This  schedule  included  interviews  with  28  client 
organizations,  the  list  of  which  is  summarized  in  Appendix  B.  Each  of  us 
interviewed  the  client  organization's  POC,  some  of  whom  brought  in  members  of 
their  staff  to  provide  input  during  the  session. 

After  the  appointments  with  the  clients  were  scheduled,  the  Client  Liaison 
Officer  provided  us  a  list  containing  the  names  and  addresses  of  the  clients  to  be 
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interviewed.  Shortly  thereafter,  we  contacted  each  client  by  phone  to  personally 
introduce  ourselves  and  to  explain  the  nature  of  our  research.  Following  these  phone 
calls,  we  mailed  each  client  a  letter  containing  the  purpose  of  the  interview,  in 
particular,  to  obtain  their  views  regarding  the  factors  that  they,  as  clients,  would 
use  to  evaluate  the  services  provided  by  their  NARDAC.  A  sample  letter  can  be 
found  in  Appendix  C.  A  week  before  the  interviews,  we  again  called  each  client  to 
verify  receipt  of  the  letter  and  to  confirm  the  date  and  time  of  our  interview.  This  call 
also  gave  the  client  the  opportunity  to  ask  specific  questions  about  the  interview  or 
our  research  and  allowed  for  any  last  minute  rescheduling  of  the  appointment  if 
required. 

To  ensure  efficient  use  of  time  during  the  interview  and  to  facilitate  data 
collection/recording,  forms  were  used  which  we  had  prepared  beforehand  to  guide  us 
through  the  interview.  These  forms  served  to  assist  us  in  collecting  background 
information  regarding  the  clients  (see  Appendix  D)  and  their  views  on  what  particular 
factors  contributed  to  their  satisfaction  with  NARDAC  services  (see  Appendix  E).  In 
addition,  the  forms  were  used  to  ensure  consistency  in  the  types  of  information  asked 
of  the  client.  In  certain  instances,  interviews  were  taped  but  not  before  receiving 
permission  from  the  client  In  no  instance  was  permission  denied. 

2.  Factor  Validation 

During  the  interviews,  we  asked  each  client  to  comment  on  the  nine  factors 
which  we  had  previously  compiled  during  the  process  described  in  Section  B  and  to 
indicate  whether  each  factor  influenced  their  satisfaction  with  the  NARDAC's 
performance  in  providing  services.  The  client  was  also  asked  to  identify  any  other 
factors  which  were  not  included  in  the  original  set  and  did  influence  their  state  of 
satisfaction.  We  then  requested  each  client  to  rank  all  the  factors  by  order  of 


importance.  In  some  cases,  clients  were  strongly  inclined  to  group  several  factors 
equally  important  under  the  same  ranking.  Other  clients  felt  that  some  of  the  factors 
were  not  pertinent  to  their  current  requirements  and  therefore  excluded  those  factors 
from  their  ranking. 

Following  the  completion  of  all  the  interviews,  we  compiled  a  tabulated  list 
of  factors  that  were  ranked  important  to  client  satisfaction  by  each  client.  The 
compiled  list  provided  us  an  across  the  board  ranking  of  all  the  factors  to  help  in 
determining  which  factors  were  considered  most  important  by  the  clients.  To  obtain 
a  composite  ranking  of  the  factors,  we  averaged  the  rankings  given  by  the  clients  on 
each  factor.  The  composite  ranking  of  the  top  six  factors  included: 

Ranking  Factor 

1  Timeliness  and  quality  of  products  and  services 

2  Technical  support 

3  System  design  and  programming  expertise 

4  Ease  of  obtaining  services 

5  Customer  support 

6  Cost  of  ADP  services 

We  ascertained  that  the  highest  ranked  factor  was,  in  fact,  two  separate 
factor  dimensions.  Timeliness  and  Quality.  By  incorporating  these  dimensions  as 
separate  factors,  a  modified  list  of  seven  factors  was  obtained.  The  remaining  factors 
in  the  original  set  were  not  discounted.  They  were  determined  to  be  too  specific  in 
scope  and  were  therefore  considered  to  be  candidate  subelements  (items)  relating  to 
one  of  the  seven  factors. 
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3.  Item  Development  and  Validation 

Our  next  step  was  to  identify  specific  items  obtained  from  the  interviews 
which  related  to  the  NARDAC's  performance  and  which  were  measurable.  The 
information  was  extracted  from  client  comments  documented  on  the  form  we  used 
during  the  interviews  (see  Part  4  of  Appendix  E)  and  obtained  in  taped  sessions. 
Working  with  our  own  individual  set  of  interview  data,  we  placed  each  item  on  a  3 
by  5  card.  When  this  was  completed,  we  compared  our  sets  of  items  for  redundancy 
and  clarification,  while  referring  to  information  from  other  surveys  and  relevant 
literature.  Our  sets  were  then  consolidated  into  a  set  containing  forty-two 
performance-related  items  which  were  then  categorized  under  one  of  the  seven 
factors  we  had  identified  in  the  previous  process.  In  those  instances  when  items 
could  not  be  placed  under  a  category  because  of  the  factor's  specificity,  we  redefined 
the  scope  of  the  factor  and  broadened  its  applicability  to  include  the  item.  The 
factors,  as  modified  by  this  process,  are  listed  below. 

•  Timeliness  of  Products  and  Services 

•  Quality  of  Products  and  Services 

•  Technical  Proficiency 

•  Hardware,  Software  and  Communications  Technology 

•  Accessibility 

•  Customer  Support 

•  Service  Level  Management 

The  final  set  of  items  and  modified  factors  established  the  client  satisfaction 
domain  for  the  NARDACs  and  provided  the  fundamental  basis  of  what  to  measure  in 
order  to  gauge  the  level  of  an  individual  client's  satisfaction.  The  key  elements  of 
this  set,  which  is  presented  in  Appendix  F  and  discussed  in  the  following  sections, 
formed  the  foundation  for  the  construction  of  the  measurement  instrument. 


D.  MEASUREMENT  INSTRUMENT 

Several  different  measuring  instruments  were  investigated  through  literature 
searches  and  reviewing  surveys  currently  being  used  in  the  government  and  private 
industry  to  determine  which  instrument  would  best  suit  our  study,  Buzzell,  Cox  and 
Brown  assert,  the  more  structured  the  measuring  instrument  is,  the  more  accurate  the 
output  [Ref.  26].  Typically,  in  marketing  measurements,  it  is  important  to  have  a 
measure  of  the  degree  of  the  response.  To  aid  in  this  type  of  measurement,  degrees 
of  response  are  scaled  to  make  more  or  less  discriminations.  Three  of  the  most 
common  marketing  measurements  are  variants  of  the  Semantic  differential  scale, 
Likert  scales  and  Paired  comparisons  (see  Figure  3.1).  According  to  Buzzell,  Cox 
and  Brown,  however. 

Although  it  is  possible  to  attach  numbers  to  such  a  verbal  scale  for 
purposes  of  analysis,  it  should  be  remembered  that  this  is  still  an 
ordinal  scale,  strictly  speaking,  and  estimates  with  regard  to  intervals 
or  degrees  of  response  are  simply  estimates.  Quantify  them  if  you 
will,  but  there  is  no  direct  means  of  determining  the  validity  of  such 
quantifications. 

There  are  several  methods  of  collecting  data  for  the  instrument.  The  most 
common  ones  used  in  marketing  research  are  personal  interviews,  telephone 
interviews  and  self-administered  questionnaires  (mail  surveys).  Listed  in  Figure  3.2 
below  are  some  of  the  advantages  and  disadvantages  of  each  method,  as  cited  by 
([Ref.  27]  and  [Ref.  28]).  The  surveys  that  we  found  currently  being  used  in  the 
government  and  industry  were  all  self-administered  questionnaires.  In  the 
government,  cost  and  time  restraints  seem  to  make  self-administered  mail 
questionnaires  more  prevalent. 

I.  Questionnaire  Development  Standards 

Due  to  the  limited  resources  available  at  a  NARDAC  to  conduct  surveys, 
self-administered  questionnaires  would  be  the  easiest  to  implement.  There  are 
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SEMANTIC  DIFFERENTIAL  SCALE 


"Would  you  rate  General  Motors  as  being: 
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LIKERT  SCALE 

"Ajax  is  an  excellent  cleanser." 

_  Agree  extremely  strongly 

_  Agree  fairly  strongly 

_  Agree 

_  Undecided 

_  Disagree 

_  Disagree  fairly  strongly 

_  Disagree  extremely  strongly 


PAIRED  COMPARISONS,  e  g.. 

Do  you  prefer  Brand  A  or  Brand  B? 
Do  you  prefer  Brand  C  or  Brand  A? 
Do  you  prefer  Brand  B  or  Brand  C? 


Figure  3.1  Commonly  Used  Types  of  Rating  Methods  and  Scales. 


1 .  Personal  interviews  : 


Advantages: 

•  They  tend  to  allow  the  use  of  a  more  representative  sample. 

•  Achieve  higher  return  rate. 

•  Produce  fewer  incomplete  questionnaires. 

•  More  questions  can  be  asked. 

•  More  complex  measurement  methods  can  be  used. 

•  Verification  or  responses  may  be  made  more  readily. 

Disadvantages: 

•  More  cosdy. 

•  Subject  to  interviewer  bias,  error  and  cheating. 

•  Subject  to  response  bias. 

2.  Telephone  interviews  : 

Advantages: 

•  Can  be  conducted  quickly. 

•  Relatively  low  cost. 

Disadvantages: 

•  Sample  bias. 

•  Usually  must  be  brief. 

3.  Self -Administered  questionnaires : 

Advantages: 

•  Least  costly. 

•  Avoid  interviewer  bias. 

•  Larger  number  of  respondents  can  be  reached. 

•  More  convenient  to  the  respondent. 

•  Requires  a  smaller  staff  for  administering. 

Disadvantages: 

•  Sample  is  almost  certainly  not  likely  to  be  "representative"  unless  follow 
up  is  done. 

•  Must  be  carefully  designed  and  pretested  to  avoid  confusion  on  the  part 
of  the  respondent. 


Figure  3.2  Data  Collection  Methods  (Advantages  and  Disadvantages) 


several  methods  of  conducting  a  self-administered  questionnaire.  The  questionnaire 
could  be  administered  to  a  group  of  respondents  gathered  at  the  same  place  at  the 
same  time  (e.g.,  at  a  NARDAC  Client  Council/Advisory  Board  meeting).  They 
could  also  be  hand-delivered  at  the  respondent's  office  to  be  completed  and  picked  up 
at  a  later  time.  Mailing  is  another  option,  or  a  combination  of  mailing  and  one  of  the 
above  methods  can  also  be  used.  [Ref.  23:  p.  159]  The  questionnaire  can  be  hand- 
delivered,  where  it  could  be  explained  and  any  questions  answered,  and  then  mailed 
when  completed.  The  questionnaire  designed  for  the  NARDACs  was  based  on  the 
premise  that  the  survey  would  be  mailed  with  limited  intervention  required  by  the 
NARDAC.  The  following  criteria  was  used  in  developing  the  NARDAC  Client 
Satisfaction  questionnaire: 

•  Instructions  for  completing  the  questionnaire  must  be  clear  and  concise. 

•  Scales  must  be  easily  understandable  and  unambiguous. 

•  Questions  will  be  closed-ended  and  as  easy  to  answer  as  possible. 

•  Definitions  will  be  provided  as  required. 

•  Comment  section  will  be  provided. 

•  The  number  of  questions  should  be  held  to  an  absolute  minimum. 

•  Questionnaire  must  have  an  overall  professional  appearance  (i.e.,  neat  and 
legible). 

Since  the  majority  of  the  questionnaire  respondents  are  considered  middle 
to  upper  level  management  within  their  organization  (GS-12  to  GM-15),  we  felt  a 
self-administered  questionnaire  would  provide  us  with  the  needed  information  with 
minimum  intrusion  on  their  schedules.  In  this  regard,  we  felt  it  was  important  that 
the  instructions  were  self-explanatory  and  the  questions  easily  answered.  According 
to  Arlene  Fink,  "A  self- administered  questionnaire  that  is  hard  to  read  can  confuse  or 
irritate  respondents.  The  result  is  a  loss  of  data."  [Ref.  28:  p.  44]  We  tried  to 


formulate  the  questions  in  the  respondents'  own  vernacular.  Definitions  of  certain 
terminology  and  phrases  were  provided  to  help  alleviate  any  ambiguities.  We  also 
felt  it  was  important  to  provide  the  respondent  with  an  opportunity  to  comment  on 
any  of  the  items  in  the  questionnaire  or  provide  additional  information  since  the 
questions  were  closed-ended.  Additionally,  the  questionnaire  had  to  make  a  good 
visual  impression.  Since  the  questionnaires  are  distributed  to  valued  NARDAC 
clients,  we  felt  that  if  the  appearance  made  a  bad  first  impression,  then  response  rate 
could  possibly  suffer. 

2.  Questionnaire  Format 

The  questionnaire  measures  the  respondents  level  of  satisfaction  with  respect  to  a  set 
of  forty-two  performance  related  items.  These  items  are  grouped  together  into  seven 
categories  or  factors  as  described  in  section  B  (Factor  Identification)  above. 
Although  we  were  concerned  with  the  length  of  the  questionnaire,  we  allowed 
adequate  space  between  questions  for  comments  and  to  prevent  the  questionnaire 
from  looking  cluttered.  The  questionnaire  is  eight  pages  long  or  four  pages  copied 
back-to-back.  It  is  widely  acknowledged  in  the  social  research  community  that  an 
improperly  laid  out  questionnaire  can  not  only  confuse  the  respondents,  but  also 
make  the  scoring  of  the  questionnaire  more  error-prone  and  time  consuming.  One  of 
the  most  common  questionnaire  formats  is  one  where  the  respondent  is  asked  to 
check  one  response  from  a  series  [Ref.  27:  p.  205].  Each  item  is  scored  using  a  five- 
point  Likert  scale.  A  not  applicable  box  is  also  provided.  According  to  Babbie, 
boxes,  adequately  spaced  apart,  are  the  best  for  the  respondent  to  answer.  The  scale 
is  measured  from  one  to  five,  where  five  is  very  satisfied ,  and  one  is  very 
dissatisfied.  The  respondents  are  asked  to  place  an  X  in  the  box  which  most 
appropriately  describes  their  level  of  satisfaction  with  that  item.  The  scales  are  laid 
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out  vertically  to  the  side  of  each  item  to  facilitate  scoring  of  the  questionnaire.  Key 
words  and/or  phrases  were  underlined  in  each  item  to  help  focus  the  respondent’s 
attention  on  the  major  point  the  item  is  measuring  (see  Appendix  F). 

3.  Questionnaire  Critique 

After  the  first  draft  of  the  questionnaire  was  completed,  it  was  sent  out  for 
review  to  all  nine  NARDAC  Client  Liaison  Officers/Marketing  representatives,  and  to 
various  Naval  Postgraduate  School  faculty  (see  Appendix  C).  Generally,  the 
comments  from  the  NARDACs  were  favorable.  Most  felt  we  had  developed  a 
comprehensive  questionnaire  and  had  addressed  the  major  issues  presently 
concerning  the  NARDAC  clients.  The  majority  of  the  comments  focused  on  the 
format  and  length  of  the  questionnaire.  Although  some  of  the  NARDACs  indicated 
there  were  some  questions  that  might  be  redundant,  none  of  them  indicated  which 
ones  were  the  culprits.  One  NARDAC  felt  the  sentence  structure  of  the  items  was 
too  choppy  and  short,  and  another  suggested  we  include  an  overall  rating  question  at 
the  end  of  the  questionnaire.  Surprisingly,  the  majority  of  the  comments  regarding 
item  construction  came  from  the  Naval  Postgraduate  School  faculty.  It  was  pointed 
out  that  many  of  our  items  were  asking  the  respondent  to  rate  more  than  one  aspect  in 
a  single  item.  For  example,  in  item  twelve,  the  respondent  is  asked  to  rate  the  format 
and  the  quality  of  the  Chargeback  Report.  The  respondent  may  feel  that  the  format  is 
satisfactory,  but  the  quality  is  not.  Other  comments  dealt  with  the  lack  of  definitions 
regarding  the  meaning  of  certain  terms,  and  the  order  in  which  the  questions  were 
presented.  The  above  comments  were  incorporated  into  the  second  draft  of  the 
questionnaire.  Comments  referring  to  the  implementation  of  the  questionnaire  were 
disregarded  because  they  fell  outside  the  scope  of  our  study  (e.g.,  adding  an  internal 
routing  stamp). 
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4.  Questionnaire  Second  Draft  (Short  Form ) 

The  second  draft  of  the  questionnaire  looked  considerably  different  from 
the  first  version  (see  Appendix  G).  It  is  divided  into  three  major  sections.  The  first 
section  still  measures  the  respondents  level  of  satisfaction  with  respect  to  a  set  of 
performance  related  items,  but  the  three  comment  lines  have  been  dropped,  two 
additional  questions  were  added,  and  the  ordering  and  grouping  of  the  questions 
were  changed.  These  revisions  made  the  questionnaire  appear  to  be  shorter  -  four 
pages  long  or  two  pages  copied  back-to-back  ( Short  Form  ).  The  items  are  still 
grouped  together  into  seven  categories  or  factors,  but  the  names  have  changed. 
Although  the  length  of  the  questionnaire  was  a  concern,  we  did  allow  adequate 
spacing  between  questions  to  prevent  the  questionnaire  from  looking  cluttered.  The 
response  boxes  formed  a  matrix  down  the  right  side  of  the  page  to  facilitate  the 
scoring  of  the  questionnaires.  The  scale  was  repeated  at  the  top  of  each  page  for  the 
convenience  of  the  respondent. 

A  new  section  was  also  added  to  the  Short  Form.  In  Robert  Alloway's 
paper.  Defining  Success  for  Data  Processing,  he  discovered  that  in  the  six  companies 
that  he  studied,  all  were  violating  the  fundamental  rule  of  management,  "...identity 
which  activities  are  most  important  and  allocate  resources  to  ensure  good 
performance  on  those  activities."  [Ref.  29:  p.  1]  Thus,  the  second  section  of  the 
questionnaire  measures  the  relative  importance  of  each  of  the  seven  major  categories 
described  in  the  first  section  of  the  questionnaire.  Again,  each  category  is  scored 
using  a  five-point  interval  scale.  Boxes  are  provided  for  the  respondents  answers. 
The  scale  is  measured  from  one  to  five,  where  five  is  critical,  and  one  is  irrelevant. 
The  use  of  the  second  section  will  be  described  in  detail  in  Chapter  V.  The  data  from 
this  section  will  not  be  used  in  the  computation  of  the  index;  it  will  only  be  used  as  a 


management  tool.  The  last  section  of  the  questionnaire  permits  the  respondent  to 
make  comments  about  any  item(s)  or  general  comments  about  the  service  provided 
by  the  NARDAC.  The  trailing  edge  of  the  questionnaire  provides  the  NARDAC  with 
pertinent  administrative  information  regarding  the  respondent. 

5.  Final  Review  of  Questionnaire 

A  revised  long  form  1  and  the  short  form  were  sent  out  to  all  NARDAC 
Commanding  Officers  (COs)  and  NAVDAC  for  one  last  review  before  the  pilot 
survey  was  conducted.  During  a  NARDAC  Board  of  Directors  meeting,  the  COs 
decided  that  the  scale  identifiers  at  each  extreme  did  not  demonstrate  enough 
differentiation  from  the  other  identifiers.  Therefore,  the  words  extremely  satisfied 
and  extremely  dissatisfied  replaced  the  identifiers  very  satisfied  and  very 
dissatisfied.  They  also  changed  the  middle  identifier  from  Neither  Satisfied  Nor 
Dissatisfied  to  Neutral.  They  felt  this  wording  would  be  be  easier  to  interpret  by  the 
respondents  and  NARDAC  management.  Other  changes  that  were  made  included  the 
definition  of  factors  and  items,  the  rearranging  and  regrouping  of  items,  and  the 
revision  of  some  of  the  factor  names.  One  additional  factor  was  also  added.  The 
short  form  of  the  questionnaire  was  unanimously  approved.  They  felt  that  the  long 
form  would  be  too  overwhelming  for  most  of  their  clients.  However,  they  still 
would  like  to  use  the  long  form  for  specific  clients  from  whom  more  data  is  required 
to  help  identify  and  resolve  discrepancies. 


1  The  revised  long  form  is  the  original  questionnaire  with  only  content  changes 
made.  The  format  remained  the  same  (see  Appendix  H). 


IV.  QUESTIONNAIRE  EVALUATION 


This  chapter  describes  the  methods  we  employed  to  evaluate  the  measurement 
instrument,  the  client  satisfaction  questionnaire.  As  discussed  in  the  preceding 
chapters,  this  instrument  was  developed  to  capture  the  domain  of  client  satisfaction 
for  the  NARDACs  and  the  distinguishable  elements  (items)  that  influence  this 
domain.  The  administration  of  the  questionnaire  to  a  group  of  subject  clients  at  a 
pilot  site  and  the  evaluation  of  the  questionnaire  based  on  the  scored  results  are 
presented.  The  results  were  analyzed  using  several  statistical  software  packages. 
STATGRAPHICS,  a  personal  computer  (PC)  based  statistical  graphics  system,  and 
the  Statistical  Package  for  Social  Scientists  (SPSS-X),  a  mainframe  computer 
program,  were  used  in  performing  descriptive  statistics,  construct  validation  tests, 
and  reliability  tests.  In  addition,  various  microcomputer  programs  were  augmented 
to  provide  other  tools  for  the  evaluation  of  the  measurement  instrument. 

A.  PILOT  DESCRIPTION 

To  collect  the  data  for  evaluation  of  the  questionnaire,  we  used  a  self- 
administered  mail  survey.  One  NARDAC  site  was  chosen  to  pilot  the  measurement 
instrument,  consisting  of  fifty-two  client  organizations  which  formed  the  sample 
population.  Although  we  realize  that  a  nationwide  survey  would  provide  us  with  a 
greater  sample  size  to  item  ratio  and  more  statistically  sound  test  results,  time  and 
schedule  constraints  and  the  lack  of  physical  resources  compelled  us  to  preclude 
exercising  this  alternative.  Nevertheless,  we  felt  the  pilot  sample  size  was  sufficient 
for  an  initial  evaluation  of  the  instrument.  Considering  geographical  proximity,  the 
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clients  serviced  by  NARDAC  San  Francisco  were  therefore  selected  as  the  subject 
group  for  evaluation  of  the  questionnaire  (see  Appendix  I). 

1 .  Pilot  Administration 

After  receiving  approval  from  the  Commanding  Officer  of  NARDAC  San 
Francisco  to  conduct  the  pilot,  we  obtained  a  list  of  active  clients  from  his  Client 
Liaison  staff.  Information  from  this  list  included  the  name  of  the  client  organization, 
name  of  the  organization's  POC,  mailing  address  and  telephone  number. 
Additionally,  the  staff  supplied  us  with  supplemental  profile  information  on  each 
client.  The  profile  data  consisted  of  current  fiscal  year  revenues  earned  to  date  from 
the  clients  and  types  of  services  provided  to  the  clients  by  the  NARDAC. 

Once  the  NARDAC  Board  of  Directors  gave  us  the  approval  to  proceed 
with  the  pilot  testing,  we  mailed  each  client  a  pilot  survey  package  that  consisted  of  a 
cover  letter,  the  self-administered  questionnaire  (see  Appendix  J)  and  a  pre¬ 
addressed  return  envelope.  In  the  cover  letter,  we  requested  each  client’s  cooperation 
in  completing  the  questionnaire  to  assist  us  in  evaluating  the  measurement 
instrument.  The  client  was  also  asked  to  critique  the  questionnaire  in  terms  of 
content,  presentation  and  format.  A  sample  of  the  cover  letter  is  presented  in 
Appendix  C. 

The  survey  packages  were  posted  to  the  clients  ten  days  before  the 
requested  return  deadline  which  allowed  the  respondents  at  least  one  full  work  week 
to  complete  and  return  the  questionnaire.  The  return  envelope  was  provided  to 
accelerate  the  return  time. 

2.  Pilot  Response 

Of  the  fifty-two  survey  packages  mailed,  we  received  twenty-six  completed 
questionnaires  which  seemed  to  suggest  a  fifty  percent  response  rate.  Current  survey 


research  guidelines  on  percentage  return  rates  indicate  that  fifty  percent  is  an 
acceptable  response  rate.  Babbie  suggests  that  a  response  rate  of  seventy  percent  or 
more  is  very  good,  a  response  rate  of  at  least  sixty  percent  is  good  and  a  response 
rate  of  at  least  fifty  percent  is  adequate  for  analysis  and  reporting.  He  does  caution, 
however,  that  this  yardstick  is  only  a  rough  guide  which  has  no  statistical  basis.  The 
accepted  practice  in  computing  response  rates  is  to  omit  all  those  questionnaires  that 
could  not  be  delivered  due  to  bad  addresses  and  the  like.  This  number  is  subtracted 
from  the  initial  sample  size  to  obtain  the  net  sample  size.  Then,  the  number  of 
completed  questionnaires  is  divided  by  the  net  sample  size  to  produce  the  response 
rate.  [Ref.  23:  p.  165]  Although  the  number  of  questionnaires  we  received  was  far 
below  the  number  we  expected,  we  felt  that  this  could  have  been  attributed  to  the 
timing  of  the  pilot  survey  which  occurred  during  the  Christmas  season  (early 
December).  To  substantiate  reasons  for  adjusting  the  net  sample  size,  it  was 
necessary  for  us  to  obtain  concrete  evidence  for  the  number  of  omitted  non¬ 
responses.  To  accomplish  this,  we  made  numerous  attempts  to  contact  by  phone 
those  clients  who  had  not  responded.  In  many  cases,  we  were  unsuccessful  in  our 
efforts  due  to  busy  signals  or  no  answer.  In  those  few  instances  when  we  were  able 
to  contact  the  client  organization,  messages  were  left  to  the  organization's  POC,  yet 
no  return  calls  were  ever  received.  Some  of  the  client  organization  POCs  who  were 
successfully  contacted  indicated  that  they  never  received  the  survey  package.  In  one 
specific  case,  the  survey  package  had  been  addressed  to  the  client  organization 
headquarters  rather  than  to  the  field  activity  where  the  organization’s  POC  was 
located.  Among  the  non-respondents  contacted,  we  accounted  for  three  clients  who 
did  not  receive  the  survey  package.  Two  survey  packages  were  returned  for  reasons 
of  insufficient  address  or  unknown  addressee.  And  lastly,  one  completed 


questionnaire  arrived  four  weeks  after  our  analysis  of  the  initial  set  of  twenty-six 
responses.  Our  net  sample  size  after  adjustment,  therefore,  was  forty-six  which 
yielded  a  response  rate  of  fifty-seven  percent. 

Due  to  time  and  schedule  constraints,  subsequent  mailings  were  not 
conducted.  Moreover,  we  determined  that  the  response  rate  of  fifty-seven  percent 
was  acceptable  for  purposes  of  our  evaluation. 

3.  Processing  of  Pilot  Data 

The  twenty-six  returned  questionnaires  were  processed  in  the  following 
manner.  As  survey  packages  were  returned  and  after  each  envelope  was  opened,  the 
questionnaire  was  reviewed  and  checked  against  the  master  mailing  list.  Each 
completed  questionnaire  was  then  assigned  a  unique  client  code.  The  convention 
used  in  assigning  the  client  code  was  straightforward  -  a  number  that  represented  the 
sequence  in  which  the  questionnaire  was  received.  The  scored  results  of  each 
questionnaire  were  then  entered  into  a  microcomputer  based  spreadsheet  (EXCEL) 
and  saved  on  diskettes  for  subsequent  processing  by  various  statistical  software 
programs.  The  specific  software  programs  and  the  results  obtained  from  the 
processing  of  the  scores  arc  discussed  in  the  next  following  sections. 

B.  DESCRIPTIVE  STATISTICS 

Basic  descriptive  statistics  were  used  *o  evaluate  the  data  collected  from  the  pilot 
survey.1  Frequency  distributions  were  performed  on  each  item  in  order  to  uncover 
any  irregularities  in  the  responses.  What  was  found,  was  an  unusually  large  number 
of  not  applicable  (N/A)  responses  had  been  given  for  many  of  the  items.  Figure  4. 1 

1  The  descriptive  statistics  developed  from  the  pilot  survey  are  proprietary 
information  of  NARDAC  San  Francisco,  and  will  not  be  disclosed  in  this  thesis. 


shows  a  histogram  of  all  the  items  and  their  associated  percent  of  not  applicable 
responses.  A  three-step  process  was  used  to  determine  the  cause  of  the  high  N/A 
response  rate. 

The  first  step  was  to  isolate  those  items  that  had  an  unusually  large  number  of 
not  applicable  responses;  the  mean  was  chosen  as  the  break-point.  Any  item  which 
had  a  not  applicable  response  rate  above  31.4  percent  was  coded  with  an  asterisk  on 
the  histogram  as  shown  in  Figure  4.1  (i.e.,  items  5,  6,  9,  17,  18,  19,  20,  21,  22, 
23,  25,  26,  27,  29,  38,  and  42).  Table  4.1  shows  the  break-down  of  the  isolated 
items  by  factor. 

It  was  interesting  to  note  that  all  of  the  items  for  the  factor  Quality  of  Products 
and  Services,  and  three  out  of  the  five  items  for  the  factor  Timeliness  of  Service,  fell 
above  the  mean  (3 1 .4%)  N/A  response  rate.  Problems  discovered  in  item 
construction  for  these  two  factors  will  be  discussed  in  Section  D  of  this  chapter. 

The  next  step  was  to  see  if  there  were  any  relationships  between  the  sixteen  items 
listed  in  Table  4.1  and  the  respondents  which  contributed  significantly  to  the  N/A 
response  rate  for  those  items.  Figure  4.2  shows  the  number  of  N/A  responses  for 
the  sixteen  isolated  items  by  respondent.  Again,  the  mean  was  chosen  as  the  break¬ 
point  to  separate  those  respondents  that  had  a  significantly  large  number  of  N/A 
responses.  The  asterisks  in  Figure  4.2  indicate  which  respondents  fell  above  the 
mean  (9)  rate  (i.e.,  respondents  1,  2,  3,  4,  5,  7,  8,  11,  12,  13,  21,  22,  23,  24,  and 
25). 

During  the  last  step,  we  looked  at  the  profile  of  the  respondents  isolated  in  the 
previous  step  to  determine  what  may  have  caused  these  respondents  to  answer  N/A  to 
so  many  items.  Table  4.2  lists  all  the  pilot  respondents  (by  code),  NARDAC  San 
Francisco's  fiscal-year-to-date  (FYTD)  revenue  for  that  respondent,  and  the  type  of 


TABLE  4.2  Pilot  Client  Profile  Data 


CLIENT  CODE 

FYTD  REVENUE  ($000) 

SERVICE  PROVIDED 

1 

23 

Risk  Assessment 

2 

5 

Training 

3 

2 

Study 

4 

1 

Programming 

5 

0 

- 

6 

46 

Training  &  Procurement 

7 

2 

Training 

8 

17 

Training 

9 

10 

IRC  Support/Sperry 

10 

5 

Labor 

11 

14 

Study  &  Procurement 

12 

1 

Programming 

13 

117 

ADP  Security  Risk 

Assesment 

14 

26 

Programming 

15 

24 

IRC  Support,  Procurement,  Tech 

Support 

16 

172 

Programming,  Telecom,  Key- 

Entry,  Burroughs,  IV  Phase 

17 

6 

FYPR-Sperry 

18 

34 

Sperry,  Comlines,  Terminal 

19 

1 

Sperry 

20 

5 

Labor,  Mi  sc,  Telecom 

21 

21 

Procurement,  Training 

22 

8 

IRC  Support,  Procurement 

23 

293 

Programming,  Procurement 

24 

6 

Training 

25 

0 

• 

26 

17 

IRC  Support,  Procurement 

Mean 

32.9 

For  all  respondents 

Mean 

7.7 

(Except  for  respondents  13 

&  23) 

Sum 

856 

For  all  respondents 
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service  provided.  The  respondents  in  bold  type  are  the  ones  isolated  in  step  two. 
The  following  observations  were  made: 


•  The  mean  FYTD  Revenue  for  all  the  isolated  respondents  (except  respondents 
13  and  23)1  was  $7,700.  As  compared  to  the  total  mean  for  all  respondents, 
these  are  relatively  small  clients. 

•  Forty  percent  of  the  highlighted  respondents  only  received  training  as  a 
service.  The  questionnaire  does  not  accommodate  training  services  as  well  as 
other  services  provided  by  the  NARDAC.  This  problematic  area  will  be 
addressed  in  Section  D  of  this  chapter.  It  should  also  be  noted  that  many  of  the 
NARDACs  distribute  separate  surveys  for  clients  receiving  training. 

•  The  contract  for  respondent  3  had  not  begun  at  the  time  the  pilot  was 
conducted. 

•  The  contracts  for  respondents  5  and  25  have  been  cancelled. 


TABLE  4.1  Break-down  of  Isolated  Factors. 


1  Respondent  13  had  a  one  time  contract  for  ADP  Security  Risk  Assessment  and  had 
little  interaction  with  NARDAC  San  Francisco.  Respondent  23  had  a  large  hardware 
procurement  which  accounted  for  75%  of  his  total  FYTD  revenue.  !t  was  felt  that  these 
two  respondents'  relatively  high  FYTD  revenue  was  not  the  norm  and  were  disregarded  in 
the  computation  of  the  mean  for  the  isolated  respondents. 


42 


It  would  be  premature  to  make  any  widespread  conclusions  about  the  quality  of 
the  questionnaire  from  this  phase  of  the  evaluation  since  the  number  of  respondents 
(26)  from  the  pilot  survey  was  small.  Further  testing  of  the  instrument  with  a  larger 
sample  size  will  be  required  in  order  properly  evaluate  the  questionnaire;  however,  it 
is  clear  that  there  are  problems  with  some  of  the  items  in  the  factors  Quality  of 
Products  and  Services  and  Timeliness  of  Services.  These  issues  will  be  addressed  in 
Section  D. 

Generally,  the  comments  about  the  questionnaire  were  quite  favorable.  Only 
seven  out  of  the  twenty-six  respondents  actually  commented  on  the  construction  of 
the  questionnaire.  Of  those  who  commented,  over  seventy  percent  thought  the 
questionnaire  was  well  prepared.  A  typical  comment  was,  "Your  questionnaire  was 
quite  thorough,  easy  to  read,  and  concise.. .the  instructions  were  fine...".  Only  one 
respondent  (#12)  felt  the  questionnaire  was  too  long,  but  he  is  a  new  client  and 
answered  thirty-one  of  the  forty-four  items  on  the  questionnaire  with  a  not  applicable 
response. 

C.  VALIDATION  AND  RELIABILITY  PROCEDURES 

The  next  step  in  the  evaluation  of  the  instrument  was  to  test  the  questionnaire  for 
its  validity  and  reliability.  In  this  study,  we  examined  content  validity  or  face  validity 
and  construct  validity.  Cronbach's  alpha  was  used  to  examine  reliability. 

1 .  Content  Validation 

Content  validation  is  determined  from  the  content  and  operations  of  the 
measurement  instrument  [Ref.  15:  p.  156].  Others,  such  as  Babbie,  refer  to  content 
validity  as  face  validity  or  logical  validity.  For  instance,  if  we  were  indeed  interested 
in  measuring  client  satisfaction  of  those  clients  supported  by  the  NARDACs,  then 
from  a  logical  point  of  view  each  of  the  items  considered  should  appear  on  its  face  to 
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indicate  the  respondent's  satisfaction  with  the  services  provided  by  the  NARDACs. 
[Ref.  23:  p.  256]  The  content  of  the  measurement  instrument,  therefore,  has  to  be 
designed  to  include  all  important  items  or  attributes  affecting  a  client’s  satisfaction 
with  the  NARDAC  support.  As  described  in  Chapter  III,  this  has  been  the 
underlying  theme  of  our  methodology  to  identify  those  items  and  to  test  the  list  of 
items  for  completeness. 

The  items  were  examined,  prior  to  the  pilot,  by  independent  groups  of 
experts  as  the  initial  phase  in  validating  the  completeness  of  the  items  list.  This 
follows  a  similar  approach  used  by  Pearson  in  which  independent  assessments  and 
the  collective  experience  of  expert  groups  provided  different  perspectives  to 
counterbalance  any  significant  omissions  that  could  have  occurred.  [Ref.  16:  p.89] 
The  first  group  of  experts  was  selected  on  the  basis  of  their  experience  in  ADP  and 
interaction  with  the  clients.  This  group  was  composed  of  all  the  NARDAC  Client 
Liaison  Officers  who  reviewed  the  initial  draft  of  the  questionnaire.  The  same 
questionnaire  was  reviewed  by  another  group  of  experts  in  the  academic  environment 
with  research  and  consulting  experience  in  the  fields  of  data  processing,  survey 
research  and  statistics,  and  organizational  management.  This  second  group  was 
comprised  of  faculty  in  the  Information  Systems,  Operations  Research  and 
Administrative  Sciences  curriculums  at  the  Naval  Postgraduate  School.  Based  on  the 
comments  and  recommendations  from  the  first  two  groups  of  experts,  the  item  list 
was  modified  and  the  questionnaire  revised.  The  revised  questionnaire  was  then 
reviewed  by  a  third  group  of  experts  with  extensive  experience  in  the  data  processing 
field  and  management  in  the  Navy  and  with  a  history  of  formal  interaction  with  client 
organizations.  This  last  independent  body  of  reviewers  was  made  up  of  the 
Commanding  Officers  of  the  NARDACs  and  the  NAVDAC  staff,  in  essence,  the 
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NARDAC  Board  of  Directors.  Details  of  these  reviews  are  described  in  Chapter  III, 
Section  D,  Parts  3  and  5.  The  modifications  recommended  by  the  three  review 
groups  and  the  independent  assessment  of  the  completeness  of  the  item  list  indicated 
that  the  important  items  influencing  a  client’s  satisfaction  with  NARDAC  services 
had  been  identified  and  validated. 

The  other  aspect  of  content  validity  deals  with  the  logical  soundness  of  the 
operations  for  measuring  the  content  of  the  items.  To  establish  the  operations  for  the 
measurement  of  this  content,  the  Likert  Scale  methodology  was  used.  The  selection 
of  this  methodology  for  appropriateness  and  the  development  of  the  measurement 
instrument  is  described  in  Chapter  III.  On  the  basis  of  this  approach,  we  determined 
that  this  measurement  process  w<s  further  evidence  of  content  validity,  although 
subjective  in  nature. 

Content  validity  was  likewise  performed  by  the  clients  themselves  during 
the  pilot  phase  of  the  measurement  instrument.  The  clients  participating  in  the  pilot 
were  asked  to  critique  the  questionnaire  by  providing  an  assessment  of  the  following: 

•  Clarity  of  item  phraseology 

•  Appearance  of  item  redundancy 

•  Omission  of  critical  items 

•  Readability  of  the  survey  and  ease  of  completion 

•  Clarity  of  the  instructions 

Their  assessment  of  the  above  items  addressed  the  questionnaire's  content  structure 
and  completeness  of  items;  therefore,  further  evidence  of  content  validity  had  been 
achieved. 
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2.  Construct  Validation 

In  the  previous  section,  content  validity  of  the  measurement  instrument  and 
the  process  in  performing  this  test  for  validity  was  described.  Construct  validity 
focuses  on  the  nature  of  the  items  being  measured  and  the  extent  to  which  these 
particular  items  relate  to  one  another.  Thus,  it  attempts  to  measure  the  correlations 
among  many  independent  items  to  determine  whether  these  items  are  strongly  enough 
related  to  describe  a  particular  relational  concept.  In  this  case,  construct  validation 
was  used  to  determine  how  strongly  each  of  the  forty-four  items  related  to  one 
another  and  to  each  of  the  eight  factors.  Factor  analysis  was  the  technique  employed 
in  an  effort  to  perform  this  type  of  validation. 

a.  Factor  Analysis  Description 

Factor  analysis  is  a  multivariate  statistical  tool  that  analyzes 
interrelationships  among  many  items  (e.g.,  questionnaire  responses)  and  then 
explaining  these  items  in  terms  of  their  common  underlying  dimensions  (factors) 
[Ref.  31:  p.  427].  It  is,  in  effect,  a  simultaneously  processed  item  interdependence 
technique  that  tells  which  item  responses  measure  the  same  factor  and  to  what  extent 
they  measure  these  factors. 

In  examining  the  pilot  responses  for  construct  validity,  we  performed 
factor  analysis  on  the  forty-four  item  scores  for  each  of  the  twenty-six  respondents. 
This  was  accomplished  using  the  factor  analysis  software  module  in 
STATGRAPHICS  on  a  Zenith  248  microcomputer  and  the  SPSS-X  factor  analysis 
program  on  an  IBM  3033/4381  computer.  The  purpose  of  these  trials  was  to 
determine  if  there  was  indeed  some  logical  pattern  among  the  forty-four  items 
intercorrelations  and  to  see  what  dimensional  factors  would  be  generated  based  on 
the  intercorrelations  of  these  empirical  input.  STATGRAPHICS  allows  the  use  of 


the  original  item  responses  and  prompts  for  the  number  of  factors  to  extract  in  the 
analysis.  SPSS-X  performs  similarly  and,  as  an  addition,  provides  the  feature  of 
generating  the  number  of  factors  automatically. 

b.  Factor  Analysis  Procedures 

The  forty-four  item  scores  for  each  of  the  twenty-six  respondents  were 
entered  into  a  file  for  processing  by  the  statistical  software  programs.  In  the  case  of 
STATGRAPHICS,  these  entries  were  simultaneously  stored  by  the  software 
program  to  the  work  diskette.  Parameter  specifications  used  for  the  factor  analysis 
runs  were  Pairwise  (to  handle  N/A  items)  and  Varimax  rotation.  The  output  in 
Pearson  Coefficient  format  (available  on  SPSS-X  only)  contained  more  meaningful 
information  for  evaluation.  The  interested  reader  is  referred  to  the 
STATGRAPHICS  User  Guide  [Ref.  32]  and  SPSS  User  Manual  [Ref.  34]  for  more 
details  on  the  various  factor  analysis  parameter  options. 

c .  Interpretation  of  Factor  Analysis  Output 

The  output  of  a  factor  analysis  program  is  presented  in  matrix  form  and 
consists  of  several  key  components.  The  columns  represent  the  factors  (artificial 
dimensions)  generated  from  the  observed  relations  among  items.  The  values  under 
each  factor  column  represent  the  correlations  between  each  item  and  each  factor  and 
is  referred  to  as  the  factor  loadings.  In  examining  the  output,  one  can  determine  the 
meaning  of  a  given  factor  based  on  those  items  that  load  highly  on  it.  Babbie  points 
out  that  the  generation  of  factors,  however,  has  no  reference  to  the  meaning  of  the 
item,  only  their  empirical  associations.  Furthermore,  he  offers  two  important  criteria 
to  consider  while  evaluating  this  data. 


•  A  factor  must  explain  a  relatively  large  proportion  of  the  variance  found  in  the 
items,  and 

•  Every  factor  should  be  more  or  less  independent  of  every  other  factor.  [Ref. 
23:  p.  328] 

Although  the  interpretation  of  the  complex  interrelationships  found  in 
the  factor  analysis  matrix  output  is  no  simple  matter,  the  following  provides  a  step- 
by-step  method  for  evaluating  the  data.1 


1 .  Examine  the  factor  matrix.  Each  column  denotes  a  separate  factor.  The  values 
beneath  the  columns  are  the  factor  loadings  for  each  item  on  each  factor.  The 
numbers  on  the  left  margin  of  the  matrix  represent  each  of  the  44  items  in  the 
questionnaire. 

2.  Begin  the  analysis  by  starting  at  the  first  item  on  the  first  factor  and  move 
horizontally  from  left  to  right,  looking  for  the  highest  loading  for  that  item  on 
any  factor.  For  sample  sizes  less  than  100,  the  lowest  factor  loading  to  be 
considered  significant  would  be  ±  .50.  If  the  highest  loading  is  significant, 
underline  it. 

3.  Proceed  to  the  second  item  and,  again  moving  from  left  to  right  horizontally, 
look  for  the  highest  loading  for  that  item  on  any  factor  and  underline  it. 
Continue  the  procedure  for  each  item  until  all  the  items  have  been  underlined 
once  for  their  highest  significant  loading  on  a  factor.  Some  items  may  have 
several  loadings. 

4.  Identify  items  that  have  not  been  underlined  (those  that  do  not  load  on  a  factor). 
If  the  item  is  considered  important,  leave  the  item  as  is.  If  the  item(s)  are 
considered  of  minor  importance  to  client  satisfaction,  the  item(s)  may  be 
eliminated  and  derive  another  factor  analysis  solution  with  the  non-loading 
items  eliminated. 

5.  When  all  significant  items  are  loading  on  a  factor  indicating  that  a  factor 
solution  has  been  obtained,  assign  a  name  or  meaning  to  the  pattern  of  factor 
loadings.  Items  with  higher  loadings  are  considered  more  important.  The  final 
result  will  be  a  label  or  a  name  that  represents  each  of  the  derived  factors.  [Ref. 
34:  pp.  250-251] 


1  It  is  recommended  that  the  expertise  of  a  social  researcher  be  used  in  interpreting  the 
factor  analysis  output. 
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d.  Pilot  Results 

The  factor  analysis  execution  runs  of  the  pilot  data  using  both 
STATGRAPHICS  and  SPSS-X  produced  unusable  output  and  therefore 
inconclusive  results.  One  explanation  for  these  results  could  be  the  small  sample  size 
to  item  ratio  for  this  pilot  (.59  :  1).  Hair  et.  al.  emphasizes  that,  as  a  general  rule, 
there  should  be  five  times  as  many  responses  as  there  are  items  (5:1)  to  be  analyzed 
and  that  this  ratio  is  considered  to  be  somewhat  conservative.  He  further  adds  that 
when  dealing  with  smaller  sample  sizes  and  a  lower  sample  size  to  item  ratio,  any 
findings  should  be  interpreted  with  caution.  [Ref.  34:  pp.  250-251]  Ideally  then,  a 
preferred  sample  size  to  ensure  a  more  meaningful  and  empirically  sound  evaluation 
of  the  questionnaire  should  be  at  least  220  (i.e.,  5  times  44  items). 

3.  Reliability  Testing 

According  to  Kerlinger,  "reliability  can  be  defined  as  the  relative  absence  of 
errors  of  measurement  in  a  measuring  instrument".  In  other  words  it  is  the  accuracy 
or  precision  of  a  measuring  instrument.  [Ref.  30:  p.  405]  There  are  basically  two 
types  of  reliability  tests  that  can  be  performed:  test-retest  method,  and  the  internal 
consistency  method  (Cronbach's  alpha).  According  to  Carmines  and  Zeller,  test- 
retest  method  is  one  of  the  easiest  ways  to  estimate  the  reliability  of  empirical 
measurements  [Ref.  31 :  p.  37].  In  this  method,  the  same  survey  is  given  to  the  same 
set  of  respondents  after  a  period  of  time,  and  the  correlation  between  the  scores  is 
obtained.  If  exactly  the  same  results  are  obtained  on  the  two  administrations  of  the 
survey,  the  test-retest  reliability  coefficient  will  be  1 .00  (i.e.,  perfect  reliability).  The 
problem  with  this  method  is  that  the  respondent’s  perception  of  client  satisfaction 
will  presumedly  change  over  time  based  on  the  quality  of  services  provided  by  the 
NARDAC.  Thus,  a  low  test-retest  reliability  correlation  may  not  indicate  that  the 
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reliability  of  the  questionnaire  is  low.  The  test-retest  method  can  also  be  expensive 
and  impractical  to  administer. 

A  much  better  method  of  determining  reliability  is  the  measurement  of  the 
amount  of  error  in  the  instrument.  The  amount  of  error  in  a  measure  can  be 
determined  using  Cronbach's  alpha  test  applied  to  interitem  correlations,  which  can 
be  expressed  as  follows: 


Np' 

[1  +  P'(N  -  1)] 


where 


N  =  the  number  of  items 

p'  =  the  mean  interitem  correlations. 


The  value  a  will  vary  between  .00  and  1.00,  when  the  mean  interitem 
correlations1  are  between  zero  and  one.  The  value  a  at  1 .00  is  perfect  reliability,  but 
a  reliability  score  of  .80  is  considered  acceptable  for  basic  research  [Ref.  15:  p.788]. 
As  with  the  factor  analysis  procedure,  the  reliability  test  was  performed,  but  because 
of  the  small  sample  size,  the  results  were  inconclusive.  It  is  recommended,  however 
that  a  reliability  test  be  performed  on  the  instrument  prior  to  implementing  the  Client 
Satisfaction  survey.  The  ratio  of  sample  size  to  number  of  items  should  be  at  least 
(5:1)  or  greater.  Reliability  testing  is  not  available  in  the  STATGRAPHICS  program 
however;  it  can  be  easily  tested  using  procedure  RELIABILITY  on  SPSS-X.  The 
Alpha  model  (Cronbach's  alpha)  is  the  default  model  in  procedure  REU ABILITY. 


1  To  find  the  mean  interitem  correlation,  the  correlation  coefficients  are  summed  and 
divided  by  the  total  number  of  coefficients. 
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D.  REVISED  QUESTIONNAIRE 

The  frequency  distributions  calculated  in  section  B  above  highlighted  two  factors 
that  displayed  an  usually  high  rate  of  not  applicable  responses:  Quality  of  Products 
and  Services  and  Timeliness  of  Services.  Each  item  within  these  two  factors  were 
carefully  examined  and  compared  to  the  profile  of  the  respondents  which  answered 
not  applicable  to  those  items.  From  our  analysis,  we  concluded  that  clients,  such  as 
those  receiving  training  or  Information  Resource  Center  (IRC)1  assistance,  had 
difficulty  answering  the  questions  in  those  two  factors. 

As  a  result,  questions  18  and  28  were  identified  as  being  too  specific  and  were 
revised  to  broaden  their  applicability.  Figure  4.3  illustrates  the  changes  made.  Of 
course,  further  testing  of  the  instrument  with  a  larger  sample  size  will  have  to  be 
performed  to  further  refine  the  instrument,  but  we  feel  these  preliminary  changes  will 
reduce  the  number  of  not  applicable  responses  for  the  factors.  Quality  of  Products 
and  Services  and  Timeliness  of  Services.  See  Appendix  K  for  the  revised 
questionnaire. 


1  The  IRC  at  NARDAC  San  Francisco  was  created  to  help  Navy  commands  and 
other  government  agencies  to  deal  with  the  microcomputer  revolution.  Basically,  it  helps 
the  client  take  full  advantage  of  the  productivity  enhancement  capabilities  of  a 
microcomputer. 
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QUALITY  OF  PRODUCT  AND  SERVICES 

(NARDAC's  ability  to  deliver  reliable  services  and  excellent  products) 


CURRENT:  18.  The  quality  of  NARDAC's  data  processing  services. 

REVISED:  18.  The  quality  of  NARDAC's  services. 

(Services  include  training,  Information  Resource  Center  (IRC) 
assistance,  data  processing.  Client  Liaison  services,  studies,  etc.) 


TIMELINESS  OF  SERVICES 

(NARDAC's  ability  to  be  punctual  and  "schedule  conscientious"  with  its  products 
and  services) 


CURRENT:  28.  Timeliness  of  deliverables  from  NARDAC. 

REVISED:  28.  Timeliness  of  services  from  NARDAC. 

(Services  include  training,  Information  Resource  Center  (IRC) 
assistance,  data  processing.  Client  Liaison  services,  studies,  etc.) 


Figure  4.3  Changes  to  items  18  &  28. 


V.  DATA  ANALYSIS  TECHNIQUES 


A.  INDEX  CONSTRUCTION 

An  index  is  a  composite  measure  very  frequently  used  in  social  research  [Ref. 
27:  p.  361  ].  Babbie  lists  the  following  advantages  of  an  index: 


•  Allows  a  researcher  to  develop  a  composite  measure  of  variables. 

•  A  single  data  item  might  not  have  enough  categories  to  provide  the  desired 
range  of  variation,  but  an  index  formed  from  several  items  would. 

•  Indexes  are  efficient  data  reduction  devices  :  several  indicators  may  be 
summarized  in  a  single  numeric  score. 

An  index  is  constructed  by  accumulating  scores  assigned  to  individual  attributes. 
The  data  from  the  client  satisfaction  questionnaire  was  formulated  into  an  index, 
which  would  empirically  describe  the  level  of  client  satisfaction.  The  Likert  scale 
was  quantified  by  assigning  values  5,  4,  3,  2,  and  1  to  extremely  satisfied,  satisfied, 
neutral,  dissatisfied,  and  extremely  dissatisfied  respectfully;  the  not  applicable  scores 
were  disregarded  in  constructing  the  index.  According  to  Babbie,  "the  Likert  format 
lends  itself  to  a  rather  straightforward  method  of  index  construction.  Since  identical 
response  categories  are  used  for  several  items  intended  to  measure  a  given  variable, 
each  such  item  can  be  scored  in  a  uniform  manner."[Ref.  27:  p.  375]  Using  the 
values  above,  the  scores  on  the  questionnaire  were  averaged  for  each  factor  to  arrive 
at  a  factor  index: 


where 


Fj  =  individual  factor  index 

rrij  =  total  number  of  items  for  factor  j 


L  =  number  of  applicable  responses 
I,j  =  numeric  score  for  item  k  of  factor  j, 

=  1,  2,  3,  4,  5. 

To  determine  the  overall  Client  Satisfaction  Index  (CSI),  all  of  the  factor  indices  (Fj) 
are  averaged: 

1  8 

CSI  =  vf  X  Fj  where 
i=l 

N  =  number  of  meaningful  factors 

(i.e.,  factors  where  at  least  one  item  was  applicable) 
i  =  the  number  of  factors. 

The  result  will  be  a  CSI  value  between  one  and  five.  Each  factor  was  given  an  equal 
weight  because  there  were  no  substantiating  reasons  for  differential  weighting  of  the 
factors.  An  interpretation  of  the  index  is  shown  in  Table  5.1. 


TABLE  5.1. 


CSI  RANGE  VALUES 

INTERPRETATION 

4.21  to  5.00 

Extremely  Satisfied 

3.41  to  4.20 

Satisfied 

2.61  to  3.40 

Neutral 

1.81  to  2.20 

Dissatisfied 

1.00  to  1.80 

Extremely  Dissatisfied 

B .  IMPORTANCE-PERFORMANCE  GRID 

The  Importance-Performance  Grid  was  developed  by  Robert  Alloway,  from  the 
Sloan  School  of  Management,  Massachusetts  Institute  of  Technology,  to  help 
identify  which  client  satisfaction  factors  are  most  important  and  to  allocate  resources 


to  ensure  good  performance  on  those  factors  (i.e.,  "prioritizes  management's 
attention").  [Ref.  29] 


The  last  section  of  the  questionnaire  is  used  to  collect  data  which  can  be  used  to 

develop  an  importance-performance  grid.  The  respondent  is  asked  to  rate  the  relative 

importance  of  each  of  the  eight  performance  related  factors  on  a  scale  of  one  to  five. 

These  importance  scores  are  then  plotted  against  the  average  performance  scores  for 

each  factor  (Fj).  After  all  the  factors  have  been  plotted,  the  grid  is  arbitrarily  divided 

into  four  quadrants.  The  placement  of  the  axes  is  a  managerial  decision  based  on 

what  senior  management  feels  are  achievable  standards  of  success  (e.g.,  see  Figure 

5.1).  Alloway  defines  the  quadrants  as  follows: 

The  upper-right  comer  is  the  relative  success  quadrant.  This  implies 
no  change  is  necessary  in  importance-performance  for  these  criteria. 

The  lower-left  quadrant  is  also  OK  in  the  sense  that  lower  performance 
on  these  comparatively  unimportant  criteria  is  acceptable.  My 
recommendation  for  these  two  quadrants,  leave  them  be,  might  appear 
much  too  benign  until  one  contemplates  the  level  of  managerial 
attention  and  effort  required  to  improve  the  lower-right  quadrant. 

The  lower-right  quadrant  is  the  real  killer.  These  criteria  have  high 
importance  but  low  performance.  These  are  the  criteria  which  min  a 
DP  department's  reputation,  drive  users  up  the  wall,  seriously  impair 
DP's  ability  to  deliver,  and  prevent  user  managers  from  receiving  their 
relevant  information. 

The  upper-left  quadrant  should  receive  declining  management 
attention.  Any  increased  efforts  to  improve  performance  here  are  a 
relative  waste  of  resources.  Clearly  DP  management  should  not  steal 
resources  from  the  success  quadrant  for  use  in  the  killer  quadrant, 
rather,  DP  should  reallocate  from  waste  and  OK  to  killer.  [Ref.  29: 
pp.  15-17] 

Each  client  will  have  a  unique  profile  based  on  his  Importance-Performance  Grid 
that  will  focus  management's  attention  on  the  factors  that  the  client  feels  requires 
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greater  attention  by  the  NARDAC.  It  will  also  assist  the  NARDAC  in  managing 
scarce  resources. 


I 


In  the  above  example  (Figure  4.1),  the  grid  clearly  illustrates  how  the  NARDAC 
performed  on  the  various  factors  and  the  relative  importance  of  each  of  those 
factors1.  Using  Alloway's  definition  of  the  quadrants  as  stated  above,  the  NARDAC 
is  performing  quite  well  in  the  area  of  System  Resources  (2)  and  Staffing  (7),  which 
are  viewed  as  important  to  the  client's  mission.  Therefore,  no  change  in  the  resource 
allocation  is  necessary.  Accessibility  (1)  and  Responsiveness  (6),  however,  rated 


1  These  scores  are  fictitious  and  were  chosen  for  illustrative  purposes  only. 
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Figure  5.1  Importance-Performance  Grid  [Ref.  29] 
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very  poorly  in  performance.  This  rating,  on  face  value,  would  indicate  a  need  by 
management  to  focus  more  attention  in  those  two  areas;  however,  this  particular 
client  views  the  two  factors  as  relatively  unimportant.  Therefore,  it  would  be  more 
productive  to  concentrate  on  the  more  important  factors.  Quality  of  Products  and 
Services  (4)  and  Timeliness  of  Services  (5).  These  two  factors  were  rated  high  in 
importance  but  low  in  performance  and  should  have  management’s  highest  attention. 
According  to  Alloway,  resources  should  be  reallocated  in  order  to  bring  factors  in  the 
killer  quadrant  into  the  success  quadrant.  He  suggests  pulling  resources  out  of  the 
waste  quadrant.  In  this  area.  Cost  Management  (3)  and  Timeliness  of  Services  (5) 
rated  high  in  performance  but  low  in  importance,  therefore  efforts  to  improve 
performance  would  be  wasting  scarce  resources. 

C.  FREQUENCY  HISTOGRAM 

Another  interesting  and  informative  way  to  view  the  data  from  the  questionnaire 
is  through  the  use  of  frequency  histograms  (Figure  5.2).  Histograms  are  a  very 
good  and  quick  visual  tool  that  can  be  used  to  see  how  all  the  respondents  answered 
a  particular  item  or  factor.  The  frequency  histogram  Fills  in  the  holes  where  the  CSI 
leaves  off.  To  illustrate  this  point,  it  is  plain  to  see  from  Figure  5.2  that  the  clients 
are  quite  satisfied  with  the  accessibility  of  the  NARDAC  with  sixty-five  percent  of 
the  respondents  answering  satisfied  or  higher.  The  histogram  could  also  point  out  a 
poorly  constructed  item  if  there  was  a  very  large  percentage  of  not  applicable 
responses. 
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VI.  CONCLUSIONS  AND  RECOMMENDATIONS 

A.  CONCLUSIONS 

In  conclusion,  the  research  objectives  of  this  study  were  accomplished.  Client 
satisfaction  was  defined  within  a  domain  of  eight  factors  made  up  of  forty-four 
attributes  (items).  An  instrument  (questionnaire)  was  developed  to  measure  these 
factors  and  an  index  was  constructed  to  derive  an  empirical  measure  of  client 
satisfaction. 

Through  the  process  of  conducting  personal  interviews  with  clients,  holding 
discussions  with  NARDAC  management,  and  digesting  numerous  articles  and 
journals  relating  to  customer  satisfaction,  client  satisfaction  was  adequately  defined 
by  eight  factors.  Forty-four  items  were  developed  to  further  define  the  factors.  A 
self-administered  mail  questionnaire  was  developed  to  collect  the  data.  This  method 
of  data  collection  was  viewed  to  be  the  easiest  and  least  costly,  as  far  as  time  and 
personnel  resources  are  concerned,  to  implement  by  the  NARDACs.  Due  to  time 
and  resource  constraints,  the  survey  instrument  was  piloted  at  one  site,  NARDAC 
San  Francisco,  using  their  fifty-two  clients  as  the  sample.  Twenty-six  respondents 
returned  the  questionnaire,  which  yielded  a  return  rate  of  fifty-seven  percent  after 
accounting  for  undelivered  questionnaires.  Next,  the  pilot  data  was  examined  using 
descriptive  statistics.  As  a  result  of  this  phase  of  the  questionnaire  evaluation,  it  was 
discovered  that  two  factors  exhibited  a  large  number  of  not  applicable  responses.  To 
alleviate  this  problem,  two  items  (18  and  28)  were  revised  to  broaden  their 
applicability.  Content  validity,  or  face  validity,  was  achieved  by  subjecting  the 
questionnaire  to  numerous  critiques.  The  critiques  were  conducted  by  the  nine 


NARDAC  Client  Liaison  Officers,  faculty  members  at  the  Naval  Postgraduate 
School,  the  nine  NARDAC  Commanding  Officers,  and  by  twenty-six  client 
organizations  during  the  pilot.  Construct  validity  and  reliability  tests  were  also 
performed,  but  due  to  the  small  sample  size  to  item  ratio  (.59  :  1),  the  results  were 
discounted.  Further  testing  with  a  larger  sample  will  have  to  be  conducted  by  the 
NARDACs  prior  to  implementing  the  survey.  Several  methods  of  analyzing  the 
survey  data  were  presented  in  Chapter  V.  An  index  was  developed  to  empirically 
represent  client  satisfaction  and  a  suggested  interpretation  of  the  index  was  provided. 
The  index  is  represented  by  the  following  equations: 


where 


Fj  =  factor  index  for  factor  j 
mj  =  total  number  of  items  for  factor  j 
L  =  number  of  applicable  responses 
I,j  =  numeric  score  for  item  k  of  factor  j, 

=  1,  2,  3,  4,  5. 

,  8 

CSI  =  X  Fj  where 
i=l 

CSI  =  Client  Satisfaction  Index 
N  =  number  of  meaningful  factors 

(i.e.,  factors  where  at  least  one  item  was  applicable) 
i  =  the  number  of  factors. 

An  Importance-Performance  grid  was  presented  to  help  NARDAC  management 
identify  which  client  satisfaction  factors  are  most  important  to  the  clients  and  how  to 
allocate  resources  accordingly  to  increase  client  satisfaction.  Another  informative 
way  to  view  the  data  collected  from  the  survey  is  to  use  frequency  histograms.  The 
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histograms  can  easily  identify  items  that  are  problematic  (i.e.,  an  item  with  a  high  not 
applicable  response  rate).  As  a  result  of  this  review,  an  item  may  need  to  be  revised 
or  maybe  even  deleted  from  the  questionnaire. 

E.  RECOMMENDATIONS 

In  this  last  section  of  our  study,  we  provide  several  strongly  recommended 
strategies  for  implementing  the  measuring  instrument  that  has  been  developed.  In 
addition,  the  results  and  conclusions  of  our  study  indicate  several  areas  that  may  be 
pursued  for  further  research. 

1.  Implementation  Strategies 

The  following  is  a  list  of  recommendations  for  measuring  and  analyzing 
client  satisfaction  for  the  Navy  Regional  Data  Automation  Centers: 

•  More  rigorous  and  extensive  validation  of  the  measurement  instrument  should 
be  conducted.  With  a  larger  sample  size  (i.e.,  at  least  220  respondents),  the 
measurement  instrument  could  be  fully  tested  and  evaluated  for  construct 
validity  using  the  factor  analysis  method  and  for  reliability  using  Cronbach's 
alpha. 

•  The  results  of  this  study  and  any  follow-on  actions  or  plans  should  be 
published  for  dissemination  to  the  NARDAC  clients.  This  information  could 
help  to  bolster  general  acceptance  of  the  instrument  by  the  client  community.  It 
is  especially  important  that  the  clients  who  participated  in  the  interview  and  the 
pilot  receive  this  feedback  to  show  confidence  that  their  comments  were  used 
constructively. 

•  The  results  of  this  study  should  also  be  provided  to  all  NARDAC  employees 
affected  by  this  study.  Such  information  would  help  to  generate  positive 
support  and  interest  in  this  area  and  foster  general  acceptance  of  the 
measurement  instrument. 


•  The  measurement  instrument  should  be  prepared  in  booklet  form  to  present  a 
more  professional  appearance.  The  booklet  is  bound  and  not  loosely  attached 
together.  Moreover,  this  is  the  norm  in  industry  and  professionally  prepared 
surveys. 

•  Responsibility  for  the  maintenance  and  administration  of  the  measurement 
instrument  and  its  associated  database  should  be  assigned  to  the  Client  Liaison 
Officer  of  each  NARDAC.  This  provides  centralized  administration  of  client 
related  data  and  the  possible  incorporation  of  the  client  satisfaction  index  (CSI) 
into  the  NARDACs  client  profile. 

•  The  instrument  should  be  initially  administered  to  all  NARDAC  clients  to 
establish  a  baseline  CSI  by  obtaining  100%  participation.  From  our  extensive 
research  and  lessons  learned  from  the  pilot,  the  questionnaire  should  be 
delivered  to  the  clients  in  person  by  a  NARDAC  representative  (e.g.,  the  Client 
Liaison  Officer/staff).  This  ensures  certainty  of  delivery  and  provides  the 
NARDAC  representative  the  opportunity  to  update  profile  information 
regarding  the  client.  The  return  of  the  questionnaire  may  be  accomplished  by 
mail  using  pre-addressed  envelopes  with  telephone  follow-up,  or  by  collecting 
the  questionnaire  in  person  on  a  predetermined  date.  To  keep  the  CSI  database 
current,  the  survey  should  be  administered  at  least  biannually  and  may  be 
conducted  after  periodic  Advisory  Board  or  Client  Council  meetings. 

•  A  cover  letter,  such  as  the  one  in  Appendix  C,  should  be  attached  in  front  of 
the  questionnaire. 

•  The  survey  results  can  easily  be  processed  and  maintained  on  a 
microcomputer.  A  microcomputer-based  spreadsheet  software  package,  such 
as  LOTUS  1-2-3 ,  can  be  used  to  consolidate  the  data  from  the  questionnaire 
and  compute  the  client  satisfaction  index.  A  spreadsheet  package  with  graphics 
capabilities  can  produce  visually  enhanced  output  of  the  data  (e.g.,  CSI 
histograms,  frequency  histograms  of  respondents,  or  other  graphics  relating  to 
the  respondents). 

•  The  measurement  instrument  should  be  used  to  track  client  satisfaction  over 
time  and  circumstance.  With  trend  analysis,  the  effects  of  technological  or 
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policy  changes  implemented  by  the  NARDACs  on  client  satisfaction  could  be 
identified  and  evaluated. 

.  Further  Research  Areas 

The  following  is  a  list  of  recommendations  for  further  research: 

The  forty-four  items  of  the  measurement  instrument  could  be  re-examined. 
Further  examination  of  the  items  could  provide  insight  into  the  characteristics 
of  each  item  which  could  further  define  client  satisfaction.  This  would  be 
beneficial  in  identifying  performance  elements  for  the  NARDACs. 

The  use  of  the  measurement  instrument  could  be  incorporated  into  a 
productivity  model  that  would  include  effectiveness  and  efficiency  measures. 
This  would  allow  for  the  setting  of  target  goals  for  each  NARDAC  and  the 
measurement  of  progress  toward  those  goals,  [adapted  from  Ref.  16:  p.  191] 

The  measurement  instrument  could  be  tailored  to  other  ADP  service 
organizations  requiring  the  measurement  of  client  satisfaction. 
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APPENDIX  A.  MAJOR  NARDAC  SERVICES 


DATA  PROCESSING  SERVICES 
TIME  SHARING 
DEDICATED 

ANALYSIS  AND  PROGRAMMING  SERVICES 
MICROCOMPUTER 
MINICOMPUTER 
MAINFRAME 

MICROCOMPUTER  TRAINING 

ADP  SECURITY  SERVICES 
TRAINING 

RISK  ASSESSMENT  ANALYSIS 
DATA  COMMUNICATIONS  PLANNING  SERVICES 
ADP  EQUIPMENT  PROCUREMENT  ASSISTANCE  SERVICES 
TECHNICAL  CONSULTATION  SERVICES 


APPENDIX  B.  CLIENT  ORGANIZATIONS  INTERVIEWED 


SAN  DIEGO: 

COMMANDER,  NAVAL  AIR  FORCE,  U.S.  PACIFIC  FLEET 
COMMANDER,  NAVAL  SURFACE  FORCE,  U.S.  PACIFIC  FLEET 

Fleet  accounting  and  disbursing  center,  u.s.  pacific  fleet 

NAVAL  AIR  STATION  MIRAMAR 

NAVAL  AIR  STATION  NORTH  ISLAND 

NAVAL  AVIATION  DEPOT  NORTH  ISLAND 

NAVAL  SEA  SUPPORT  CENTER,  U.S.  PACIFIC  FLEET 

NAVAL  STATION  SAN  DIEGO 

PERSONNEL  SUPPORT  ACTIVITY  SAN  DIEGO 

PERSONNEL  SUPPORT  ACTIVITY  NAVAL  TRAINING  CENTER 

MONTEREY: 

NAVAL  POSTGRADUATE  SCHOOL 

SAN  FRANCISCO: 

COMMANDER,  NAVAL  BASE  SAN  FRANCISCO 

MILITARY  SEALIFT  COMMAND,  U.S.  PACIFIC  FLEET 

NAVAL  AIR  STATION  ALAMEDA 

NAVAL  AVIATION  DEPOT  ALAMEDA 

NAVY  SUPPLY  CENTER  OAKLAND 

PERSONNEL  SUPPORT  ACTIVITY  SAN  FRANCISCO 

SHIP  INTERMEDIATE  MAINTENANCE  ACTIVITY  ALAMEDA 

WASHINGTON,  D.C.  : 

NAVAL  DATA  AUTOMATION  COMMAND 
NAVY  INSPECTOR  GENERAL 
CHIEF  OF  NAVAL  PERSONNEL 
COMPTROLLER  OF  THE  NAVY 

PERSONNEL  SUPPORT  ACTIVITY,  NAVAL  DISTRICT  WASHINGTON 
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NORFOLK: 


COMMANDER  IN  CHIEF,  U.S.  ATLANTIC  FLEET 

FLEET  ACCOUNTING  AND  DISBURSING  CENTER,  U.S.  ATLANTIC  FLEET 

NAVAL  AIR  STATION  NORFOLK 

NAVAL  AVIATION  DEPOT  NORFOLK 

NAVAL  FACILITIES  ENGINEERING  COMMAND 
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APPENDIX  C.  CORRESPONDENCE 


17  August  1987 


Commander 

Naval  Interview  Headquarters 
U.S.  Atlantic  Fleet 
Norfolk,  V  A  23150 

Dear  Interviewee, 

In  a  continuing  effort  to  better  serve  their  customers,  the  Naval  Data  Automation  Command 
(NAVDAC)  and  the  Navy  Regional  Data  Automation  Centers  (NARDACs)  have 
undertaken  steps  to  improve  customer  satisfaction.  As  part  of  this  effort,  they  have 
requested  our  assistance  in  developing  an  index  which  will  be  used  as  a  measure  of 
customer  satisfaction.  We  are  students  at  the  Naval  Postgraduate  School  in  the  Computer 
Systems  Management  curriculum.  We  are  conducting  this  research  in  partial  fulfillment  of 
our  Master  of  Science  Degree  in  Information  Systems  and  in  hopes  of  providing  NAVDAC 
and  the  NARDACs  with  a  meaningful  and  useful  tool. 

To  develop  the  customer  satisfaction  index,  we  must  first  identify  those  critical  factors 
which  the  customers  would  use  to  evaluate  the  services  provided  by  the  NARDAC.  As 
discussed  in  our  phone  conversation,  we  will  be  conducting  a  personal  interview  with  you 
regarding  these  customer  satisfaction  attributes  on  the  «date»  of  August  «time».  We  expect 
the  interview  to  last  approximately  one  hour.  Below,  is  a  representative  list  of  factors 
which  we  feel  relate  to  customer  satisfaction.  We  would  appreciate  your  views  on  them 
during  the  interview,  or  any  others  that  you  think  may  be  pertinent. 

•  Computer  capabilities  (mainframe/min i/micro  support) 

•  Ease  of  obtaining  services 

•  Systems  design  and  programming 

•  Technical  support 

•  ADP  consultation  and  assistance 

•  Training 

•  Cost  of  ADP  services 

•  Customer  support 

•  Timeliness  and  quality  of  products  and  services  provided 

If  you  have  any  questions  concerning  the  interview  or  our  schedule,  please  leave  a  message 
for  us  at  our  curriculum  office,  Autovon  878-2174/2175  or  commercial  (408)  646- 
2174/2175,  and  we  will  contact  you  as  soon  as  possible. 

We  appreciate  your  assistance  in  this  effort  and  look  forward  to  meeting  you  on  the  <date>. 

Sincerely, 


Prima  A.  Morris  Robert  J.  Birdwell 

LCDR,  USN  LT,  SC,  USN 
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19  October  1987 


I 


Dear  Client  Liason  Officer, 

As  part  of  the  continuing  effort  to  better  serve  their  customers,  the  NAVDAC  Board  of 
Directors  requested  our  assistance  in  developing  an  index  which  will  be  used  as  a  measure 
of  customer  satisfaction.  We  are  students  at  the  Naval  Postgraduate  School  in  the 
Computer  Systems  Technology  curriculum.  The  research  we  are  conducting  in  this  area  is 
in  partial  fulfillment  of  our  Masters  of  Science  Degree  in  Information  Systems,  and  will 
hopefully  provide  NAVDAC  and  the  NARDACs  with  a  meaningful  and  useful  tool. 

In  developing  the  customer  satisfaction  index,  we  have  identified  those  critical  factors 
which  existing  customers  would  use  to  evaluate  the  services  provided  by  the  NARDAC. 
We  obtained  these  relevant  data  through  a  series  of  personal  interviews  from  a  cross¬ 
sampling  of  customers  served  by  NARDAC  San  Diego,  NARDAC  San  Francisco, 
NARDAC  Washington  and  NARDAC  Norfolk.  These  interviews  were  conducted  in  the 
format  shown  in  enclosures  (1)  and  (2).  In  addition  to  the  interviews,  we  have  also 
conducted  comprehensive  literature  searches,  reviewed  customer  satisfaction 
surveys/questionnaires  currently  in  use  by  government  and  commercial  organziations,  and 
interviewed  a  number  of  marketing  and  client  liason  representatives  in  private  industry  to 
complement  our  research. 

We  value  your  input  and  field  expedience  in  the  client  liason  area.  Please  review  the  initial 
draft  of  the  customer  satisfaction  survey  (enclosure  (3))  and  forward  your  comments  to  us 
by  1  November  1987.  Enclosed  is  a  pre-addressed,  stamped  envelope  for  your 
convenience  and  the  timely  delivery  of  your  comments.  The  return  date  is  critical  to  our 
research  deadline;  therefore,  we  would  appreciate  your  cooperation  in  helping  us  meet  our 
milestones. 

If  you  have  any  questions  regarding  the  survey  and/or  our  research,  please  leave  us  a 
message  at  Autovon  878-2174/2175  or  commercial  (408)646-2174/2175  and  we  will  return 
your  call  as  soon  as  possible. 


Sincerely, 


Prima  A.  Morris,  LCDR,  USN 


Robert  J.  Birdwell,  LT,  SC,  USN 


13  November  1987 


Commanding  Officer 
NARDAC 


Dear  Captain, 

We  are  forwarding  two  (2)  drafts  (revised)  of  the  NARDAC  Client  Satisfaction  Survey, 
which  we  developed  in  response  to  requirements  from  the  NARDAC  Board  of  Directors, 
for  your  review  and  comments.  Both  questionnaires  are  identical  in  content  but  differ  in 
format.  As  pan  of  our  research,  please  indicate  which  questionnaire  format  (long  form  or 
short  form)  you  prefer  and  why.  As  requested  by  Commander  Taylor,  please  forward 
your  comments  via  electronic  mail  to  NARDAC  F  'arl  Harbor  by  23  November  1987. 

If  you  have  questions  regarding  the  questionnaires  and/or  our  research,  we  may  be  reached 
at  Autovon  878-2174/2175  or  commercial  (408)  646-2174/2175. 


Very  Respectfully, 


FVima  A.  Morris,  LCDR,  USN 


Robert  J.  Birdwell,  LT,  SC,  USN 
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7  December  1987 


Commander 
Naval  Client 
U.S.  Pacific  Fleet 
San  Francisco,  CA  94130 


Dear  Client, 

We  are  students  at  the  Naval  Postgraduate  School  in  the  Computer  Systems  Technology 
Curriculum  currently  developing  a  Client  Satisfaction  Index  for  the  Navy  Regional  Data 
Automation  Centers  (NARDACs).  The  Client  Satisfaction  Index  is  designed  to  measure 
each  client's  level  of  satisfaction  with  the  products  and  services  provided  by  the  NARDAC. 
In  order  to  arrive  at  this  measurement,  a  questionnaire  is  used  as  the  instrument  from  which 
the  Client  Satisfaction  Index  is  derived.  This  questionnaire  was  developed  based  on 
comments  from  NARDAC  clients  whom  we  interviewed  in  your  area  and  other 
geographical  locations. 

Please  assist  us  in  validating  the  Client  Satisfaction  Survey  instrument  by  completing  the 
enclosed  questionnaire.  We  also  request  your  personal  critique  of  the  questionnaire. 
Please  comment  on  the  following: 

1.  Are  there  any  items  you  didn't  understand? 

2.  Are  there  any  items  you  felt  were  redundant? 

3.  Are  there  any  items  you  feel  are  critical  to  client  satisfaction  that  were  not 
addressed  in  the  survey? 

4.  Was  the  survey  easy  to  read  and  complete  (format)? 

5.  Were  the  instructions  to  complete  the  survey  sufficient? 

Please  write  your  comments  directly  on  the  survey  below  the  items  you  are  addressing  or  at 
the  end  of  the  survey.  We  are  soliciting  your  personal  comments,  therefore  a  formal  reply 
from  your  command  is  not  required.  Your  comments  and  the  results  of  your  completed 
survey  will  be  held  in  strict  confidence,  to  be  used  only  by  us  to  assess  the  quality  and 
validity  of  the  survey  instrument. 

In  order  for  us  to  meet  our  scheduled  milestones,  please  return  your  completed  survey  in 
the  enclosed,  pre-addressed  envelope  as  soon  as  possible  but  not  later  than  17  December 
1987.  If  you  have  any  questions  regarding  the  survey  or  any  aspect  of  our  research 
efforts,  we  may  be  reached  at  Autovon  878-2174/2175  or  commercial  (408)646- 
2174/2175. 


Thank  you  for  your  interest  and  cooperation. 


Sincerely, 

Prima  A.  Morris,  LCDR,  USN 
Robert  J.  Birdwell,  LT,  SC,  USN 
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«NARDAC  COMMAND  LETTERHEAD» 

Date 


From:  Commanding  Officer,  Navy  Regional  Data  Automation  Center, _ 

To:  Commanding  Officer, _ ,  (Code:  XXX) 

Subj:  CLIENT  SATISFACTION  SURVEY 

Enel:  (1)  Client  Satisfaction  Survey 

1 .  As  part  of  a  continuing  effort  to  improve  our  service,  please  take  a  few  minutes  to 
complete  the  enclosed  questionnaire. 

2.  The  purpose  of  the  Client  Satisfaction  Survey  is  to  assist  us  in  determining  what 
action  can  be  taken  to  develop  more  effective  ADP  support  for  our  customers.  Your 
response  will  enable  us  to  better  understand  your  present  and  future  needs.  The 
questionnaire  is  designed  to  snapshot  present  conditions;  therefore,  please  answer  the 
questionaire  to  reflect  current  conditions. 

3.  We  feel  this  survey  will  be  mutually  beneficial.  We  appreciate  your  continued 
assistance  in  helping  us  serve  you. 


APPENDIX  D.  INTERVIEW  INFORMATION  SHEET 


DATE  OF  INTERVIEW: 

NAME  OF  ORGANIZATION: 

NAME  OF  INTERVIEWEE: 

GRADE /RANK: 

CURRENT  POSITION: 

LENGTH  OF  TIME 
INVOLVED  WITH  NARDAC: 

TYPE  OF  SERVICE(S): 
NARDAC  PROVIDES: 


FREQUENCY  OF  INTERACTION 
WITH  NARDAC  (i.e.,  daily  .weekly):. 


WT 

si 


„PJP j.j  «Ji  m  "J  u  m  m  t  »jj  ot  ».l  m  ■>  m  tot  to  »jj  ■  ji  '-• 


APPENDIX  E.  INTERVIEW  QUESTIONS 


1 .  What  are  the  factors  or  areas  you  feel  that  the  NARDAC  customers  should  use 
to  measure  customer  satisfaction? 


2.  What  is,  in  your  opinion,  a  satisfied  NARDAC  customer? 


3.  Rank  the  factors  in  order  of  importance: 

_ Computer  capabilities  (mainframe/mi  nVmicro  support) 

_ Ease  of  obtaining  services 

_ System  design  and  programming  expertise 

_ Technical  support 

_ ADP  consultation  and  assistance 

_ Training 

_ Cost  of  ADP  services 

_ Customer  support 

_ Timeliness  and  quality  of  products  and  services  provided 


4.  Specifically,  how  would  you  evaluate  or  measure  each  of  the  factors? 


£ 


,v 

■X 

£ 
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APPENDIX  F.  ORIGINAL  QUESTIONNAIRE 


|1  'tAFOAC  CUSTOAGP  SATISFACTION  SUHV&r 


;  Additional  comments  are  welcome). 


1 .  NAROAC'i  understanding  at  the  ussaQ  on  your  command  tor  lata  or  u 


2.  NARD  AC's  familiarity  with  your  command's  mission. 


3.  How  satisfied  are  you  with  tha  fc 
new  proiacts? 


i  preparing  proposals  for 


4.  Flexibility  to  satisfactonfy  respond  to  etiangss  m  your  specif  ice!  on  requirements. 


attitude  of  tfte  NAROAC  personnel  welt  whom  you  OSes 


6.  Promulgation  of  general  oumose  information  about  what  is  happening  around  NAROAC 
(l.e..  new  hardware,  system  software,  application  software,  training,  new  personnel,  etc) 
through  newsletters,  electronic  bulletin  Boards,  and  Client  Counci  Ms  songs. 


□  0  NOT  ARRJCABLE 

0  s  verysatbfbj 
D  A  satbfed 

D  3  r€fTRER  SATBFED  NOR  OtSSATlSFED 
Q  2  otssnity-HJ 

□  1  VERY  0BSATBFB3 

Q  0  NOT*peuc»a£ 

5  VERY  SATBRED 
4  SATBFHJ 

3  retTMER  satbfed  nor  otssansrcD 


t  VERY  OlSSATBFED 


a  nor*peuMB£ 

□  5  VERY  SATBFED 

□  4  SATBRB3 

in  3  NEITHER  SATBFED  NOR  OtSSATlSFED 

G  2  otssansFED 
G  1  VERY  OOSATBFED 


)  NOTAPRUCA8U 
I]  5  VERYSATBRED 
I]  4  SATBRHJ 

G  3  TCfTHER  SATBFED  NOR  OlSSATBFED 
G  2  O&SATBFCD 
G  t  VERY  OtSSATBRED 


)  NOTAPRJCASLE 
5  VERY  SATBFED 
4  SATBRED 

3  rCtTRER  SATBRED  NOR  OlSSATBFED 


1  VERY  OBIATBRED 

0  NOT  APRJCASLE 
G  5  VERY  SATBRED 
G  4  SAfBFUJ 

G  3  fEtTHER  SATBRED  NOR  OtSSATWED 
G  2  nSMTBRED 
G  1  VERY  OtSSATBRED 


t 


2  wrsMC  cusnxefi  SATSFAcnoN  survey 

7.  Timglmaaa  ot  rasoonsa  tor  a  raouaat  tor  sarvcaa 


8.  Laval  of  anarmon  grvan  to  your  oroiact  and  command. 


9.  Whan  a  orobtam  aasoaatad  with  a  NARDAC  product  or  sannca  a  raponad.  now  aatatad 
ara  you  with  tha  limalmaaa  and  quality  ot  th«  rasoluiron? 


Sarvtca  Laval  Manaaamant. 

10.  Accuracy  ot  tha  mrtial  planning/coat  aatimataa  tor  your  projact  raouaft 


11.  Ability  to  adaouataty  axplain  coata  and  yarify  chamaa. 


12.  Formal  anti  Quality  at  Chary  abac*  Rapoff. 


0  NoraanxMLE 

0  5  veRysatbred 

□  4  SUTURED 

D  3  tCtT>CRSATBflEDNOAaSS»TaFeD 

□  2  D1SSATBFED 

G  1  VERY  OSSATURED 


0  notarrucab£ 

G  5  VERY  SATURED 
G  4  SATUFBD 

G  3  (CITHER  SATUEEiD  NOR  DtSSATtSFED 

G  2 

G  i 


I  0  NOTJUWUCMLE 
G  5  VERYSATUaCD 
G  4  SATURED 

G  3  (Cn>«<  SATUFCD  NOR  OISSATUFED 
G  2  DBSATHFHJ 
G  1  VERY  DUSATURED 


I  o  notaiwucarle 
Q  S  VERY  SATURED 
G  4  SATURB3 

Q  3  (CITHER  SATUFCO  NOR  OSSATISEeD 
G  2  OSSATURED 
Q  1  VERY  OOSATURED 


G  0  nctarpucajle 

G  5  VERYSATH»*D 
G  4  satbrbj 

G  3  (CITHER  SATUECD  NOR  OOSATHFED 
G  2  OSSATURED 
G  1  VERY  OSSATURED 


G  0  NOrrtaauSMLE 
G  5  VERY  SUTURED 
G  4  SATURED 

G  3  (CrTVCR  SATURED  NOR  OSSATURED 

G  2  OBSATBRH) 

G  1  VERY  OSSATURED 


S  rttmtc  cusTottn  s<  rsfAtmoH  survey 


25.  Tha  :«cfinc*i  staff  a  wall  varsad  in  a  hmad  ranaa  ol  ADP  subtact  araaa 


0  0  K3T»pauCAa£ 

G  5  HGH.YAGREE 
G  a  tom 

G  3  icrrvcn  agree  nor  osagrs 

G  2  CBMJfB 

G  1  HGH.Y  OOAGREE 


26.  Tha  tsenncaJ  staff  a  kaaping  abraast  of  tha  lafasl  davaloomarts  at  tha  ADP  arana. 


G  o  Ncrramjcaau 

G  5  XOM.V  AGREE 

G  4  tan ft 

Q  3  *CmCR  MACE  NOS  DOAOHEE 

G  1  iaan.voaAOPEE 


27.  Ability  to  design  and  develop  craativa  and  innovatrva  tvatama. 


D  0  NCTTAPPUCAAl 

G  5  VERYSATBflED 
G  4  SATOPED 

G  3  lerTHEH  IATSTED  NOR  OaSATBEED 
G  2  OBSATUPED 
G  1  VERY  OtSSATUPED 


28.  Parsonnal  assignad  to  your  proiact  damonatratad  tha  propar  skill  laval  raquirad  to 
partorm  tha  taak(a)  apaofiad  m  tha  Statamant  of  Work. 


G  0  Norspaucau 

D  5  VERVSATOFH) 

D  4  5ATHPB3 

G  3  ICfTHER  SATBFED  NOR  DISSATtSFED 
G  2  DtSSATSPED 
G  1  VStYOBSATBFBD 


29.  Ability  of  tha  tachracal  staff  to  communieata  at  cuitomar  tarms/languaga. 


G  0  NtyrasaicaKE 
G  5  VERYEAtBPED 
G  4  &ATSPED 

G  3  NEftMBIMTWCDNtMOSaATBFED 
G  2  0WA1BPB) 

G  1  VERY  OBSATBEBD 


Quality  of  Products  and  Sarvleaa. 

30.  Tha  »"«<  «TYiifaev  of  dakvarablaa.  G  0  kjtaprlCABE 

_  G  5  VERY  SATBFED 

_ _ _ _ _ _ _  Q  4  SATBWD 

Q  3  fCfTVER  SATBFED  NOR  OtSSATBFED 
G  2  BSSATBFED 
G  t  VERY  OKU  TBEEO 


6  NARQAC  ajSTOH£R  WnSftCTVN  SURVEY 


31.  Avaiiahilffv  of  onlina  and  Batch  ayatama  (I.a..  tha  total  tima  tha  n  up). 


0  0  HOT  APPUCAJLE 
0  5  VERY  SATBEED 
0  4  SATBEED 

0  3  IdTHER  SATBEED  NOP  DISSATtSEED 
D  2  OISSATBFED 
□  1  VERY  DttSATEEED 


32.  Quality  of  ayatam  document  an&vmanuaa  (a.g..  readable.  con  act  up-to-data). 


0  o  Nor*ppuc*a£ 

D  5  VERY  SATBEED 
0  4  SATBE® 

Q  3  tEmCR  SATBEED  HOB  DtSSATBEED 

D  2  mhi am 

O  1  VERY  0BSAT8EED 


33.  Tha  performance  of  your  ayatem(a)  currently  runnng  on  NAROAC  computer*. 


□  0  notaeeuc **£ 

G  S  VERY  SATBEED 
G  4  SATBE® 

G  3  NEITHER  SATBEED  NOW  DISSATlSf  ED 
G  2  DrSSATBEE) 

G  1  VEHYOOSATBEED 


34.  NAROAC  davalODt  I 


<  program*  tor  tha*  cuatamer*. 


Q  0  NCTAimuCABE 
Q  5  MQM.Y  AGREE 
G  4  AGREE 

G  3  (EITVER  AGREE  HOR  DISAGREE 

G  2  disagree 

Q  1  MQM.Y  DISAGREE 


35.  Response  tuna  of  onlina  ayftama. 


0  notarpucaSLE 
G  5  VERYSATBEED 
G  4  SATBEED 

G  3  tEfTHER  SATBEED  NOR  CXSSATISFED 
G  2  OOSATBEED 
G  1  VERY  OSSATBEED 


36.  MftlQl  -*ng*  partormad  on  currant 
modrftcaoont  and  oonactiona). 


tystama  (La.,  raaponarvanaaa  m  incorporating 


0  NOTAPaUGAaLE 

Q  5  VERYSATBEED 

G  4  SATBEED 

G  3  AEfTHER  SATBEED  NOR  0BSAT13EED 
Q  2  DBBATBRCD 
G  1  VERY  OBSATBRED 
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7  HAPOAC  CUSTOMER  SAT5FACTON  SLWVEV 

37.  Quality  of  training. 


PatllniU  Products  ind  Ssrvlcss- 

38.  Mnomnco  to  atoioct  tcnodulon  u  to oaliod  in  too  Stotomom  nt  Work. 


39.  Adhsrancs  lo  nmdufflnn  nroesrmng  acftsdulav 


40.  Tmohnooo  ol  praams  rooora. 


41.  Timstv  notification  at  mow  a«Uv«  m  output 


42.  Rsiisbto  and  wn«»  Othvorv  ot  omnut. 


□  0  NOT  ASSUCABLE 
D  5  VERY  SATSFED 
D  4  SATBFBJ 

0  3  t*rT>«R  satbeed  nos  oissatiseed 

□  2  OCSSATBflH) 

0  1  VERY  OGSATBEED 


0  0  NOTAPPUCABU 
G  S  VBTYSATtSFED 
G  4  SATBEED 

G  3  rCrTVCR  SATBFED  NOS  DtSSATTSEED 
G  2  WSSATBES3) 

G  1  VERYOMATBEEO 

G  o  notapsucaeu 

G  5  VERYSATBEED 
G  4  SATBFED 

G  3  *rn«R  SATBFED  NOS  BSSATISFED 
G  2  0BSATBEB5 
G  1  VERY  DHSATBFED 

G  0  NBTAPFUCASU 
G  5  VERYSATBFED 
G  4  SATBEHJ 

G  3  rETTNER  SATBFED  NOS  DSSATBFED 
G  2  DOSATBEED 
G  1  VERY  OOSATBFED 

G  o  Nontmctac 

G  $  HOH.YAOREE 
G  4  AGES 

G  3  NETTVEP  AGREE  NOB  DSAGflBE 

G  2  OMOSEE 
G  1  HDHYOBAOHEE 

G  0  NOTAESUCMU 
G  5  HOM.YAGFS 
G  4  AGREE 

0  3  WnXSAOSEENOSMSOSEE 
G  1  HOM-YDOAOSS 


(Optaolt  Nsm»_ 


<KiA iraar 


ThsnA  you  tor  com  [Rating  this  survey.  Ths  andosed  omotooo  hss  Man  provxtad  tor  your  oonvsnwnos. 
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APPENDIX  G.  QUESTIONNAIRE  SECOND  DRAFT 

SHORT  FORM 


I  HAROAC  CLlgftT  U TV* ACTIO*  SUWVfY 

PLEASE  PLACE  AN  X  IN  THE  BOX  WHICH  BEST  DESCRIBES  YOUR  LEVEL  Of  SATISFACTION  WITH  EACH  OP  THE 
FOLLOWING  ITEMS,  i Additional  commgRU  may  b*  madt  or  thg  last  page). 

U9K  T>4ft  FOUIOW1MO  3CALI: 

WA  w  ApPtCAau  5  very  SA-nsna  4  satisfied  3  mettvcr  satbfcd  nor  ocssatisfcd  2  cwsAnsneo  i  very  cwsa-hsp® 


AfifilllUUiiX 


1. 

How  satistiad  ara  you  w*h  ma  jviimri'.i.itv  ra  your  NAROAC  Min  d  contact 

WaO 

sO 

aO 

£3 

£3 

U 

2. 

Tha  omcdurw  (or  Detaining  various  oroducta  and  aarviosa. 

(ratar  to  N A VOACINST  5230. 1 ) 

n/aO 

£3 

aO 

£3 

£3 

.□ 

3. 

NAROACs  cumnt  hour,  d  noanmon 

waD 

sO  aO 

£3 

£3 

n 

4. 

WaO 

c 

aO 

£1 

£3 

o 

5. 

Th«  quantity  ot  vnitor  oarltinn  soacaa  at  NAROAC. 

n/aQ 

jO 

aO 

£3 

£3 

a 

Svatam  ffaiourcaa. 

(Encampaaaaa  hamwara.  jothaana.  and  tatapneaaaatg  tachnotoqy) 

». 

(i.»..  mantrama.  mmoonwara.  memeomputara) 

hiaO 

=0 

aO 

£1 

n 

a 

7. 

NAROAC  abMy  to  maat  your  agtaGtHLdMonidad  idormaaon  syalatn  raquramwita. 

waO 

n 

aO 

£1 

£3 

o 

a. 

Tfra  currancv  and  vanatv  d  xdrwara  oacManaa  avadatda. 

waO 

& 

aO 

£3 

£3 

a 

9. 

command's  oparanonai  nsada. 

N/aQ 

50 

aO 

£3 

£3 

i0 

gflil  MmmmtL 

10. 

How  sauatiad  ara  you  wah  NAROACa  adA aranoa  to  "i«t  himn^a? 

nw»Q 

50 

aO 

£3 

£3 

i0. 

11. 

waO 

50 

aO 

£3 

£3 

o 

12. 

Tha  tonne  d  tfia  CAarvaOart  Raport  (La.  aany  to  mad  and  undatttand). 

waO 

50  40 

£3 

£1 

i0 

13. 

Tha  teoutCL  d  tha  CAargafaadt  Raport 

waO 

sO  aO  aO 

£3 

i0 

14. 

N/aO 

sO  aQ  aO 

£3 

i0 
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|  NAMuAC  CUlrfT  SATOrACTfOm  SU/tVfY 
USE  THS  FOLLOWING  SCALE: 


H/A  NETT  APNLCAflU  5  VERY  SATOTEO  4  SATBFCD  3  «TTXR  SATVFB)  NOR  OttSATIsnB)  2  OtSSATBFED  1  VERY  OSSATOTB) 

15.  Flexibility  m  cnargmg  for  —mom  (•£.,  focod-pnoo  contract,  tiarad  rata  ttructura.  ate.).  niaO  aD  3Q  2O  iQ 


(NARDAC's  mousy  to  gatw  tm fiatta  wvm  and  arcaAsnf  products) 


16.  How  sstistiad  an  you  with  tha  quality  of  t 


1 7.  Tha  quality  of  NARDAC's  ( 


1 9.  The  raiiabilitv  of  online  and  bach  tyams  ft#.,  the  local  tsna  me  ays lam  ■  up). 


20.  Tha  accuracy  of  system  documents!  wvmanuals. 


21 .  Tha  oartormanca  of  your  syaam(s)  cunantty  runmng  an  NAROAC  comouten. 


22.  Tha  response  lima  of  NARDACm  nnlina  systems. 


23.  Whan  a  Drotnam  ataooatad  with  a  NAROAC  product  or  satvioa  ■  raortad. 
ara  you  with  ma  flualllY  fll  ttl«  flMIUllfln? 


24.  How  satisfiad  an  you  with  NAROACs  adharanoa  to  BOUICLlGtiadlllat 1 

Statement  of  Wortt? 

25.  Adhannea  to  prod ucl on  orocaaamn  schedules. 


26.  Timalmass  of  omorass  reports 

27.  Softwara  maintenance  parformad  on  a  Hating  tystams  (la- 
aorraamg  program  anon). 

26.  Timaimm  nniifiannn  c«  delays  m  output. 

29.  Timalinams  Of  llaliuanhlaa 


30.  Timafinaas  of  nsoonaa  lor  a  [ 


niaO  n  n  n  n  a 

n/aO  n  n  n  n  a 

niaQ  a  n  n  n  a 

niaD  jZ  cO  aD  £1  iD 

n/aO  sD  (D  £  £1  iQ 

n/aG  £  tO  jG  £  £ 

n/aO  £  <0  £  £  £ 

d.  n/aO  £  £  £  £  £ 


n/aD  £  tO  aO  £1  £ 

niaO  fl  £  £  £  £ 

n/aQ  £  £  £  £  £ 

n/aD  sO  £  £  £  £ 

n/aD  £££££ 
n/aD  £££££ 
n/aD  ££££.□ 


|  turn me  cutMT  urmtenoH  aoavay 

USE  THE  EOtLOWWO  SCALC 

HIA  tar  WIXMU  5  VERY  SATOFED  4  SATQFED  ]  NEfTVER  SATBFED  NOB  DBt»*nsnB3  2  OBSATSFEB  1  VERY  OBSATBFED 


31. 

Whan  a  wootam  asaoctatad  with  a  NAROAC  product  or  sarvoa  a  reported,  how  saMfiad 
wa  you  wan  tna  timaunass  of  ma  rasaiutnn?. 

N/aO 

5 □ 

n 

30 

n 

tO 

ComORlRnev 

32. 

How  satisfiad  ara  you  with  tha  taotmeal  si  aft's  tavat  of  aioanisah 

fTaehmeal  jatff  nauoos  oompuim  sptcmlots,  programmer*,  end  aytfam  wiafyata  J 

N/aO 

«□ 

n 

£3 

n 

a 

33. 

N/aO 

sO 

n 

30 

n 

i0 

34. 

Tha  taenneaf  staffs  knowtadaa  of  tha  mast  davatoomants  m  thatr 
raa pactiva  A OP  fiatda. 

N/aO 

a 

£3 

£] 

n 

i0 

35 

NTaO 

o 

40 

£3 

n 

o 

36. 

Tha  NAROAC  oaraonnal  assignad  to  your  extract  (i.a..do  thay  aarnuoauasa  tha 

Nf/O 

sO 

40 

£3 

n 

,□ 

prooar  skill  IrvrI  lo  pRrtorm  tha  lask(s)  spacifiad  m  tha  Stalamam  of  Worts. 


Coordination  »nd  Communications 


3T. 

How  satisfiad  ara  you  with  NAROACs  undarstanding  of  tha  umoLon  your  command  lor 
iata.gr  inaecurata  products  or  samcas? 

N/aO 

£3 

40 

30 

20 

iO 

38. 

n/aO 

50 

40 

30 

£3 

tD 

39. 

n/aO 

£3 

aO 

£3 

aO 

iO 

40. 

Flexibility  to  aatiaf actortly  raapond  So  changaa  at  your  spaticasion  raouaamanta. 

n/aO 

50 

aO 

£3 

n 

iO 

41. 

n/aQ 

50 

aO 

£3 

a 

£3 

42. 

(!■•■■  naw  hardware.  ayaram  soffsnra.  apple arott  software,  treating,  naw  personnel.  me.) 
through  nswsiaaar*.  aiactromc  OuSewt  hoards,  and  Chant  Council  meetings. 

N/aO 

sO 

aO 

30 

n 

tO 

43. 

n/aO 

sO 

aO 

30 

n 

iQ 

44. 

(Technics/  aafft  includes  oomp Mar  spacsafisfa.  programmars,  arte  ayaram  arsafyaar; 

n/aO 

50 

aO 

30 

n 

iO 
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MAHOJkC  CUtHT  SATT3TACTKM  tUPVtT 
UM  THC  10LL0WIH0  K1U: 

N/A  NOT  IPOlCAAU  5  VERY  SATQFED  4  SATBFBD  3  NEfnCD  SlTBFCC  NOA  OBSinSFKD  2  DB3AT1SFKD  1  VERY  CXSSATSttEO 


PLEASE  PLACE  AN  X  IN  THE  BOX  WHICH  BEST  DESCRIBES  THE 
ORGANIZATION  OP  EACH  OF  THE  FOLLOWING  PERFORMANCE  t 
SATISFACTION. 


?  IMPORTANCE  TO  YOUR 
.  AS  IT  RELATES  TO  CLIENT 


USC  THI  FOLLOWtMO  SCMA: 

1.  Accessibility 

S  Cornell. 

sO 

4  NSSOfTTAMT 

n 

3  1VE0AQE 

n 

2  NOT  IMPOST  ANT 

20 

1  IRRELEVANT 

iD 

2. 

Syatam  Resources 

n 

u 

n 

£] 

,□ 

3. 

Coal  Management 

5 □ 

n 

n 

2O 

n 

4. 

Quality  ol  Products  and  Sarvloaa 

n 

n 

3 o 

£1 

£ 

5. 

Tlmallnaaa  at  Sarvloaa 

n 

n 

30 

£3 

a 

6. 

Competency 

n 

n 

3 □ 

£1 

n 

7 

Coordination  and  Communications 

£3 

a 

3D 

£1 

0 

Comments: 


Thank  you  for  completing  this  survey.  The  enclosed  envelope  has  been  provided  for  your  convenience. 


,'-Av/-  -V>VA«V>V>\ 


LN. w**  a*»  **  J|*»  »**  aT 


APPENDIX  H.  QUESTIONNAIRE  SECOND  DRAFT 

LONG  FORM 


SAPCAC  a  * NT  SATISFACTION  SUtVTY 

Please  place  aa  X  in  the  box  wtuch  beat  describes  your  nf  <nnsfnrno»  anth  e«rii  of  the  following  sunemeatt: 

(Additional  comments  ar«  weUamai. 

ACStlltfrllltV 

1.  How  MMH1  am  you  «wh  Ifc  lyailaBlIfV  a!  vnur  MABTIA^  nnH  nl  MM?  D  OwrimMI 

D  S  vbivsatw* 

_ _ _ _ _ _  D  4  UTWCO 

—  Q  3  *rr>«4MTaroNO«OBs*Ti«o 

0  2  CWMBl 

□  t  wroMinn) 


2.  Th«  □rocafluma  for  aouinna  vanoua  amaien  anfl  niMeM 
(nfr  id  NAVOAC1NST  SZSO.  1) 


a 
□ 
□ 
□ 

□  2 

□  1 


5  vsnutmi 

4  s*weo 


3  IVTVffH  UTWQ  NO*  OISSATmD 


3.  NARDACl  cunn  l<rwir«  n#  ntwwr.1 


4.  Th«  mKH  laoiinn  at  NAHOAC  Mf  m  «« nmnmmL 


s 


S  VBH1MM 
4  3*111  »0 
0  3  lenaw 

D  2  M 

D 1  vi 


ITWBJ  NOR  DISlATBreD 


5.  Th«  quantity  ct  YlMBf  MfUlfB  3MTJI  *  NAROAC. 


D  0  tar 
0  5  VCRTS41 

n !  4  j* 


8 


_  3 

□  2 

□  t 


Swum  Httnurew. 

(Enoomoattm  Ittmmi*.  laftwm.  *nt  ttbpmttmng  fcfinoloqy) 


ff.A^mmnlnnm,  Mnaompunr*.  wkneonvumnt  T 


88 


1 S.  Rmwiiiv  in  cnaiging  tor  aarntcaa  flxt}-onc*  i 


16.  How  uastwd  aro  you  Mtti  tn«  guoity  ot  NAROACs 


I  S  vearanaN 
I  a  Miaui 
I  3 


1 7.  Tha  quality  of  NARQACs 


85  vCRTUTm 
4  sat  anvil 
Q  3  •cmVAlATI 
G  2  ova  i— 
G  1  vwro«a»Tt 


$ 


□□□□□  *,□□□□□  «□□□□□  oDDCOO  *nmnn 


i>i  j*l  |«»  l Vfc'aTlU  1> . 


0  0  rrrr  tm  rm  r 

5  V0WSAWK> 

4  IATWBO 

3  f*m«*  SAWED  NON  DlSSA  WED 

2  on  wo 
^  VWfDWOT— 

rrrrmrnT 

5  vprr  sawbd 

4  SAWED 

3  P«m«N  SAWED  NON  0ISSA  WO 

2  OWWB) 

1  ErroiiATVB 


minrf  r 

5  vottsawo 

4  SA  WED 

3  femes  1AW»  NOS  OOSAWCD 


or 

5  Wtsai 
4  IA1 

3 
2 
1 


LOT< 


VJ 

& 

•J» 

i*. 


'•‘i 

1 


5] 


i 


|  mfotc  clmmt  urartcnoN  aunry 

31.  Whan  «  bibMwh  moomd  wwn  a  NABOAC  product  or  hwci  a 


raeanad.  how  aatiaftad 


32.  How  aauahaa  ai*  you  with 
(Ttcnncl  atH  metuOo*  o 


eompunr  aoaeuMa.  ptognmon.  and  syttom  tnaiyta) 


33.  Tha  tachncal  atafTs  i 


3^.  Tha  tachncal  atafTi  | 


35.  NAROACt  ability  to  Oaaqn  and  ( 


36.  Tha  HARD  AC  panonnal  aawgnad  to  your  project.  (I.a.,  do  thay  damonatrate  tha 
nrooaf  akin  la»ai  to  paftocm  tha  taa«*)  aoacrhad  in  tha  Siatamant  dt  Wont. 


0  0  rrmynrT 
D  5  v*nra»Toawj 

0  4  aafhBD 

G  3  tcroch  aaruno  nob  ooutisfe) 
D  2  nawawe 

0  t  nauuMaiaaw 


G  0  tt  *ff  r*ar 
G  5  votva*Twa 
G  a  a*  i  aiau 
G  3  icm«haaTi 

G  2  ini— 

G  t  vdrroaaan 


G  o  irrimrrar 
G  5  vaanraai* 

G  3  Kroaaat 
Q  2  aaitaa 
G  t  tcnriMaat 


G  0  Noramcau 
Q  5  wwraATwn 
G  4  tuawD 
G  3  icm«ha*Ti 
G  2  unaifeu 

0  1  vcnvoaaan 


G  o  wTwucia 

G  5  vanatti 


'  ■  ■ 


*«cv 


APPENDIX  I.  PILOT  POPULATION 


Naval  Surface  Force, U.S.  Pacific  Beet,  San  Diego 

Fleet  Accounting  and  Disbursing  Center  U.S.  Pacific  Fleet,  San  Diego 

Fleet  Hospital  Support  Office,  Alameda 

Naval  Station,Treasure  Island,  San  Francisco 

Naval  Air  Station,  Lemoore 

Naval  Air  Systems  Command,  Naval  Air  System  Command  HQ,  Washington,  D.C. 

Naval  Biosciences  Laboratory,  Oakland 

Naval  Strike  Warfare  Center,  Naval  Air  Station,  Fallon 

Shipbuilding.Con version  and  Repair  USN,  Hunters  Point,  San  Franciscco 

Naval  Weapons  Station,  Concord,  CA 

Naval  Supply  Systems  Command,  Washington,  D.C. 

SIMA  NRMF  San  Francisco,  NAS  Alameda 

USS  Enterprise  CVN-65 

Naval  Base,  San  Francisco 

Navy  Public  Works  Center  San  Francisco  Bay 

Mare  Island  Naval  Shipyard,  Vallejo 

Naval  Education  and  Training  Financial  Information  Processing  Center,  NAS 
Pensacola 

Navy  Accounting  and  Finance  Center,  Washington,  D.C 

Naval  Air  Station,  Alameda 

Naval  Reserve  Force,  New  Orleans 

Naval  Air  Station,  Moffett  Field 

Service  Group  1,  Oakland 

Shipbuilding,  Conversion  and  Repair  USN,  Seattle 

U.S.  Maritime  Defense  Zone  Pacific,  Alameda 

Navy  Military  Personnel  Command,  Washington,  D.C. 

Naval  Aviation  Depot,  Naval  Air  Station,  Alameda 
Navy  Comptroller  Standard  System  Activity,  Pensacola 
Naval  Supply  Center,  Oakland 
Personnel  Support  Activity,  San  Francisco 
Military  Sealift  Command  Pacific,  Oakland 
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I» 


,-gVi 


,'»4»aj 


fa»i 


i 


Naval  School  Physical  Distribution  Management,  Oakland 
Naval  Medical  Command,  Northwest  Region,  Oakland 
Special  Boat  Unit  Eleven,  San  Francisco 
Naval  Postgraduate  School,  Monterey 
Naval  Facilities  Engineering  Command,  Alexandria,  VA 
USS  Kiska 

Naval  Supply  Center  Puget  Sound,  Bremerton,  WA 

Joint  Military  Postal  Activity  Pacific,  San  Francisco 

Marine  Barracks,  Naval  Air  Station,  Alameda 

Sacramento  Army  Depot,  Sacramento 

Navy  Astronautics  Group,  Point  Mugu,  CA 

Chief  Preservation  Assistance  Branch,  San  Francisco 

Navy  Resale  and  Services  Support  Office,  Field  Support  Office,  Oakland 

Navy  Legal  Service  Office,  San  Francisco 

Mare  Island  Naval  Shipyard,  Vallejo 

Naval  Station  Mare  Island,  Vallejo 

Tactical  Electronic  Warfare  Squadron  One  Twenty  Nine,  NAS  Whidbey  Island 
Defense  Subsistence  Region  Pacific,  Alameda 
Naval  Air  Station,  Fallon 

Naval  and  Marine  Corps  Reserve  Center,  San  Francisco 
Naval  Electronic  Systems  Engineering  Center  Vallejo 
NARDAC  Newport 


APPENDIX  J.  PILOT  QUESTIONNAIRE 


|  MAMAC  CUtMT  UTUrtCtKtt  VMVTT 

PLEASE  PLACE  AN  X  IN  THE  SOX  WHICH  BEST  DESCRIBES  YOUR  LEVEL  Of  SATISFACTION  WITH  EACH  OP  THE 
POLLOWINC  ITEMS.  I Additional  comtmna  may  b*  madt  on  ‘At  last  pagtl. 

us*  thb  touotma  acala: 

WA  nor  «pRMU  s  ExmecLv  uraFeo  «  satur®  1  wnui  2  obsatbfib)  1  BrraEM&r  oasA-rapeo 


AfiCIEEUilitt. 
lEttt  0)  obtaining  ttnnot) 


1. 

Thd  AvtilAfuliiv  a*  your  NAROAC  boa*  at  eonttCL 

n/aO 

£3 

aO 

£ 

£ 

£ 

2. 

And  satvicm. 

iwaO 

£3 

£3 

£ 

£ 

£3 

3. 

niaO 

£1 

£3 

£ 

£ 

£ 

A. 

n/a£ 

£3 

£ 

£ 

£ 

£ 

S. 

nmO 

£1 

£ 

£ 

£ 

£ 

Syatam  RAAOufcam. 

l&Kamptmtm  amm»  aaRiw.  tad  iMiwnwin)  tteanotogy) 

8. 

(La..  mtmfrtntt.  mneomouttn.  maoaompumrt) 

waO 

sO 

aO 

£ 

£ 

£ 

7. 

waQ 

£ 

£3 

£ 

£ 

£ 

8. 

n/a£ 

£1 

£3 

£ 

£ 

£ 

9. 

Th«  ability  at  NAROA C  Id  pnxndd  AdAiyian  t«iani nmt..i  wwm  to  moat  your 
command's  opanoon*  n— dm 

nm£ 

sO 

£3 

£ 

£ 

£ 

Can  .M»n»a»mttU. 

(inaudtt  control*  avtr  pmpct  budgam.  mm  ehmgod  tor  ttnncm.  bUng  prootdunt, 
tad  pact  smxturmt) 

10. 

NAROACa  aM*v  a  prwnd*  An  aecuntt  ttotmmont  at  bath 
tfiA  co*u  And  im  senaduw  for  a  pnnaot 

niaO 

£ 

aO 

£ 

£ 

£ 

11. 

NAROACa  Adhwnod  10  asSBOAllME 

ntaO 

£ 

4J 

£ 

£ 

£ 

12. 

n,n 

£3 

£ 

£ 

£ 

£ 

13. 

NAROACa  Chwgadddt  Ptooa  (La.  taty  10  mad  tad  undtmadX 

*n 

£3 

£3 

£ 

£ 

£ 

14. 

T>»  tfCUCIQt  0*  NAROACa  CnanjAO**  Raport. 

waD 

£3 

£1 

£ 

£ 

£ 

MAKOAC  CUtMT  SATOkACTtCM  SUaVWT 


USE  TH*  KHXOWIMO  (CUI: 


N/A  not  »pm.c*au  5  ormEiCLY  satbtso  4  satbfed  3  ns/tiuu.  2  dosatoted  1  EXTHEACLY  ossatbasd 


15. 

Ni'aO 

n 

£2 

3 □ 

£2 

D 

16. 

Fl*«ibilitv  in  charging  tor  NAROAC  product*  and  Mrvcaa. 

< a.g ..  tixod-pncm  contract,  uarad  rata  ttwctura,  etc.). 

N/aD 

£2 

£2 

£2 

£2 

iD 

INARDACs  ability  to  aolivor  nuM  ton x*t  and  ascatant  produca) 

17. 

Th*  quality  o(  NAROAC  davokmad  acohcatnn  system  amount 

N/aO 

(O 

£2 

£2 

£2 

U 

18. 

n/aO 

n 

£2 

£2 

£2 

iD 

19. 

0-a..  physical  tacunty.  utar  nt/pasnoroa,  ale.) 

n/aO 

£2 

£2 

£2 

£2 

iD 

20. 

Tha  ratiataiirv  ot  NAROAC  onima  and  natch  mmh  H  a  tha  total  rma  tha  system  m  up). 

N/aO 

£2 

£2 

3 □ 

£2 

iD 

21. 

Th*  0 !  NAROAC*  *y»t am  docurnantaporvmanuala. 

n/aD 

£2 

£2 

£3 

£2 

iD 

22. 

waD 

£2 

£2 

3D 

£2 

iD 

23 

N/aD 

£2 

£2 

£2 

£2 

D 

(NARDAC't  aoiary  to  oo  punausl  and  *  schedule  eonaoeneoue*  oath  Is  products  and  sememe) 

24 

NAROAC*  adharano*  to  nmiacughaduf.  aa  apadfiad  in  tha 

Proiact  Raquaat. 

n/aO 

£2 

£2 

£2 

£2 

.□ 

25 

N/aD 

£2 

£2 

£2 

£2 

iD 

26 

N/aO 

£2 

£2 

£2 

£2 

n 

27. 

n/aO 

£2 

£2 

£2 

£2 

n 

28. 

Timatmaaa  ot  datmaraBta*  from  NAROAC. 

N/aO 

£2 

£2 

£2 

£2 

,□ 

R«»pon»lv»n*«* 

fHow  quickly  NAROAC  reacts  or  responds  »  out  needs) 
29.  SflftWlfl  mmmuiMlBl  padormad 


corraang  program  •non). 


on  aamdng  NAROAC  oyntomo  (ie~  reapontnonaas  in 


n/aD  c  c  a  n  n 


KA.V-V.VJW1I  WA  .  ■>.»  •.  -. 


|  Mimic  cutrfr  aamracnoa  tmvtr 

UM  TMt  POLLCWH40  aCAU: 

H/ A  h or  iimucmu  S  Exmamr  satbfbj  4  satwcd  3  mutml  2  i 


1  EXTOCLV  06SATBPB) 


30.  Tha  raaoonarranaaa  of  NARDAC  to  a  : 


n/aO  n  n  n  n  a 


31.  Th»  raponaivanaaa  of  NAROAC  to  laaofvmg  ■  raportad  probfam  aaaoaattd  »ah  ■  product 
or  s«rvic*. 


n/aO  sO  iO  iQ  tD 


32.  Tha  HtuhiiHY  of  NARDAC  to  changoa  w  your  profct  mwonmi 


n/aO  sD  aO  3O  2O  iD 


wn  n  n  n  n  a 


Staffing 

(Encompass**  tha  rasoureaMnaa*  and  rnounxm  quatheaum  of  NAROAC  paraofstat) 


34.  ThaovarmN 


1  tha  NAROAC  c 


N/aO  £3  aD  £3  £3  iD 


35.  Tha  l*iiiLflLi*M0aA  of  ttw  NAROAC  paraomal  »ah  whom  you  daaL 


n/aD  £3  £3  £3  £3  £3 


36.  Tha  ability  of  NARDAC  paiaonnal  10 1 
raapaaiwa  functional  vu. 


N/aD  sD  aO  :D  2D  iD 


f  tha  NAflDAC  paraonnai  aath  whom  you  OaM.  N/aO  5D  aO  3D  2O  iD 


38.  Tha  lachnical  atdfa  ability  10  Oaann  and  daaalan  or.at*.  and  mnm/atiu.  ««■.». 
(Tactmcal  Stan  netudaa  corny***  ipanalati.  pmgramman.  ana  syttam  mtalyam) 


N/aD  5D  aD  3D  2D  iD 


39.  Tha  tacfmieal  staff!  ability  10  a 


N/aO  £3  aD  aD  2O  iD 


40.  Tha  ability  of  NARDACa  laconical  staff  to  1 


(Th,  mannar  n  wmch  NARDAC  conducts  butmaas  mlh  its  dams) 


n/aO  S3  O  £1  S3  O 


41.  NARDACa  tamiliarav  with  your  command's  im««inn  ma  a  mam,  to  procaaaaig  N/aO  «0  aD  3D  2D  iD 


42.  NARDACA  undarsaanrtina  of  lha  iiw 
or  sannoaa. 

43.  Tha  coordmafion  and  toMoa>  up  on  | 


n/aO  £3  aD  3D  £3  iD 
n/aO  aD  aD  sD  jD  ,□ 


AA.  Promulgation  Of  auMCTl  BUfflOM  mtnmi«lBn  about  aaff  a  haeoaung  around  NARDAC  N//D  iD  ^  aD  £3  iD 
(ls.,  now  harowan.  syttam  to ttwara.  APpAcafon  aoftwar*.  tramaip.  naw  panormat,  ale,) 
through  nawafabara.  atachenic  OUMn  Ooanla.  and  CAanf  Count*  maaonga. 


1C 


-*»  a 


4  naatuc  clmht  unvon  auarar 


PLEASE  PLACE  AN  X  IN  THE  BOX  WHICH  BEST  DESCRIBES  THE 
ORGANIZATION  OP  EACH  OF  THE  FOLLOWING  PEOEnPUANCE  t 
SATISFACTION. 


f  IMPORTANCE  TO  YOUR 
.  AS  IT  RELATES  TO  CLIENT 


USE  TIN  ROUOWMO  SCALE: 

1.  Accessibility 

(Emu  ot  obtmrmg  turn) 

2.  System  Resources 
(Encompasses  hammers.  sotteera.  and 
teleprocessing  technology) 


3  CRITICAL  4  MSOPTAMT  3  AVERAGE  2  NOT  SWOSTAKT  1  IRRELEVANT 

sO  aG  aO  £2  iG 


Cost  Msnsgement 

( Include s  comma  over  prefect  budget*  rant 
charged  hr  services  biding  procedures,  and  pres 
ttruauraa) 

n 

n 

n 

£1 

iG 

Qusllty  at  Products  snd  Services 
INARDACa  ability  to  deliver  reliable  aenncea  and 
erceMenf  products) 

n 

n 

n 

aG 

iO 

Timeline ee  ol  Services 

1 HARD  AC's  ability  to  be  punctual  bid  ‘schedule 
conaoenooua'  mah  ta  products  and  aemoae) 

n 

n 

aG 

aG 

i0 

(How  omckty  HARD  AC  reacts  or  reeponda  as 
cient  needa) 

5 a 

n 

aG 

aG 

iG 

Stefflng 

(Enoompasaaa  the  rseourcetulnen  and  requiaae 
quetbeabona  ol  HARDAC  personnel) 

sG 

aG 

aG 

aG 

iG 

General  Business  Prsctlces 

sG 

n 

aG 

aG 

iG 

Comments! 


NeneNyoeo 


•efyewNAROACPOC_ 


Thank  you  for  complaint  this  survey.  The  enclosed  envelope  has  been  provided  for  your  convenience. 


V.V.’AV.V 


WM 


APPENDIX  K.  REVISED  QUESTIONNAIRE 


nenoee  cutxr  unmenw  sum vwr 

PLEASE  PLACE  AN  X  IK  THE  BOX  WHICH  BEST  DESCRIBES  YOUR  LEVEL  OP  SATISFACTION  WITH  EACH  OP  THE 
POLLOWINO  ITEMS.  (Additional  comments  may  be  made  on  die  last  page). 

UM  rm  FOLLOWING  5CALA: 

WA  NOT  APPUCMU  S  EXn*Ma.YMTOFCD  4  unpco  3  KUIMl  2  OSMTVe)  1  EXIPBuMLY  0CO»T13F*0 


Accaaalhlllty. 

Etta  at  ootenmq  semes) 


1 .  Tha  nt  your  HARO  AC  BOM  ol  contact 


2.  Tha  anesEUtlB  «m'«a  inNAVOAClNSr  «aa  r  lor  <*»»ninn  venous  NAROAC  atnduoa 
and  larvcaa. 


3.  Tha  currant  hound  oormon  a  NAROAC. 

4.  Tha  nhv<M~at  mw  nt  NAROAC  raiMva  la  you t  command. 

5.  Tha  Quantity  nt  mat  *  NAROAC. 


Svitam  Raaaureao- 

(Encompasses  hsrdttere.  eottmere.  end  yltpwrwm;  technology) 


S.  Tha  nonant  ryynnutar  hanty-  nananjiti^  a  NAROAC. 
(T.a.  memtreme.  t  moomputers.  mcwaomputers) 


7.  Tha  ability  <d  NAROAC  la  maal  your  m 
a.  Thacurrwtcv  and  vanatv  ol 


4  nitnmaf 


UYMm  rBPunm«m» 


l  NAROAC. 


9.  Tha  atnbty  ol  NAROAC  to  aranda  adaouala  I 
oonunand's  opantional  noada. 


1  to  maal  your 


Sail  MinsntmtnL 
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NAHOAC  CLMNT  SATatACTiaK  aunvcr 


um  t*«  wuann  ac*ii: 

NfA  NOT  AWUCMU  3  emfie.rMTMD 


3  «umN. 


1  EXTTddY  OtSSATBFED 


IS.  The  ojjliflLaaf  NARDAC  charges  tor  product*  and  serves*. 


1  g.  Flaaibilitv  I n  charging  let  NARDAC  products  am)  UMCM. 
(a.g..  taad-ono*  contract  tiarad  rata  structure,  me.). 


Quality  o)  Product*  and  Service* 

(NARDAC  *  atWay  to  mw  itairaM*  Samoa*  and  aacaianf  product*) 

17.  The  Quality  at  NARDAC  darmWrari  armlirratyin  naan 


18.  Tha  quality  ot  NARDAC*  wvrti 

(Sarvrca*  induda  training. Intormanon  Raaourca  Cantar  (RC)  aiifnca , 
processing. Client  liaiaon  aarvicaa.ttudMa.atc.) 

19.  Tha  aacumy  comrma  mrar  your  data  that  »  maaaamaa  an  NARDAC  eomniaarm. 
fta.,  pftyscaf  aacumy.  uaar  rdyMssirords.  ate.) 

20.  Tha  raliafaihtv  ot  NARDAC  onana  and  hatch  ayatama  li.a  Ola  ratal  Umalha  auaaam  it  tail 


21.  Tha  ggomot  of  NARDAC*  system  documamaoonAnanuala. 


22.  Tha  raaoonaa  ton*  cl  NARDAC*  nnUna  ayatama. 


23.  Tha  cualitv  of  tha  raanlutan  la  a  nmotaiw  aaaociatari  nath  a  NARDAC  awadum  ar  aanana 


Tlmallnaaa  ot  Sarvlcaa. 

(NARDAC  $  AtMiiry  p  oa  punctual  and  ’  achadtd*  conacranaoua*  atfi  (a  produeta  and  same**) 


24.  NAROACt  adharanoa  to  j 
Protect  Request 

25.  Tha  ability  ot  NARDAC  to  adnata  to  I 


26.  Tha  frequency  of  NARDAC  omoraaa 


27.  Tha  timely  notllicatian  of  delay*  m  output. 


28.  Timahnaaa  nt  aaminaa  hum  NARDAC 

(Sarvrca*  induda  training. Information  naoouip*  Cantor  (RC) 
orocaaamg, Client  Lrtaann  aanncaa.atudiaa.aie.) 

Rt88nn«lv9n«88 

(How  qutdttf  NARDAC  ramcn  or  raapondt  to  cmnt  naadb) 
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oomcang  program  an tori). 
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WA  NOT  tmxMLE  S  EXTBeuSL-Y  UT«B 


4  J* 


S  kutmi  2  ctnwB  i  emcke.'r  oouivbi 


31.  Tin  rcponawanaaa  at  WARD  AC  to  readying  «  raponad  pioWem  I 
or  aennce. 


32.  Tha  flBiihilitv  Of  NATO  AC  to  change*  in  your  propel  raquramara*. 

33.  Tha  ImLsLmBlIQGt  Ptwidad  in  praoanng 

aiming. 


rOuaMca 


tt*  NARDAC  ptnoml) 


34.  Thoovoran  nr  wuniu  nahai  B»  NATOAC  nmnamn. 

35.  Tha  layal  at  amaWM  at  Mia  MAWDAC  fl—MIWi  awdi  aAaan  UBU  d 


36.  The  adMy  of  NATOAC  peraonnei  to  IMO I 
reapacnve  functional  area. 


37.  Th.m..na.»aaniaa.1|1|l  — — M  f»  Mannar.  amaaa aal  .ah  «*aaa  «au  *aat 


36.  Tha  techmcal  atdT*  adMy  to  deargn  and  develop  SMUu60gjnBnBML*7M*"*- 
(Ttchnctl  staff  MUaa  computer  jpeaaAad.  programmer*.  m id  oyawm  army**; 


39.  Tha  tachmcal  atair*  adMy  to  adatacaonly  j 


IWlgflm 


40.  The  adMy  d  NATOAC*  technical  I 


fitntm  ImDium  Practlwg. 

f7ha  manner  at  add)  NATOAC  oonduco 
41.  NAROACa  tamilWIK  «Mh  your  C 


42.  Mannary.  ..~i~e»na.». «« iaa  »»»>«  «i  your  command 

Of  OtfVIOM. 


43.  Tha  coonlinahon  ana  tallcar  up  on 


jMBfml  m  nftnmnrmv 

■HU  aM  A  h  area  nan  amend  NATO  AC 
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n/aO 

£3 

£1 

£1 

£3 

£3 

Nrt£3 

£3 

£1 

£3 

£3 

i  □ 

N/«0 

£3 

£3 

£3 

£3 

£3 

N/«0 

=o 

n 

£3 

£3 

£3 

N/aO 

£1 

n 

£3 

£3 

iD 

nmO 

£1 

n 

£3 

£3 

£3 

n/aO 

£1 

n 

£3 

£3 

£3 

n/aO 

£1 

£1 

£3 

£3 

£3 

N/aO 

£3 

£3 

£3 

£3 

£3 

N/aO 

n 

£3 

£3 

£3 

£3 

N/aO 

£1 

£3 

£3 

£3 

iD 

niaG 

£3 

£1 

£3 

£3 

£J 

hiaO 

£3 

£3 

£3 

£3 

£3 

ntaO 

£] 

£3 

£3 

£3 

£3 

104 


w  wmvvv  w:  runur 


PflHff 


Titnunuravn 


Wi1^1  TT  a  7  ■  i.”  v  ww  rvr^m 


denote  cuorr  temrecnom  ttmvwr 

0LF6SAlHP^™^LLOI^e*«.^!«  0f5Cm,ES  ™*  BEJLAT1YE  fMPPffTMCf  to  you*  organisation 

OF  EACH  OF  THE  FOLLOWING  PERFORMANCE  CPITTBIA  ts  IT  RELATES  TO  CLIENT  SATISFACTION. 
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Thank  you  for  computing  this  survey.  The  enclosed  envelope  has  been  provided  for  your 


convenience. 
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